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Appendix A. About the study 


This appendix includes additional background information on early childhood education (ECE) workforce 
education and wages nationwide, ECE workforce education and wages in Oregon, national initiatives to increase 
ECE workforce education, career lattices and workforce registries, and Oregon’s workforce registry and career 
lattice. In addition, it provides a review of the research literature on the implementation and effectiveness of 
information campaigns and financial incentives related to the interventions used in this study. 


Education and wages of the early childhood education workforce across the country 


While nearly all states have identified a core set of competencies and skills required for the ECE workforce, there 
is no consensus across states on how those competencies and skills translate into minimum education 
requirements or credentials or on the requirements for different types of workplaces (for example, home-based 
care versus centers; Whitebook et al., 2016). As of 2016, only 11 states had a minimum credential or vocational 
requirement for ECE professionals who are employed outside public preschool systems (Oregon is not one of the 
11); most states require only a high school diploma and, in several cases, some additional training (Whitebook et 
al., 2016). These minimal requirements do not reflect what the science of child development suggests ECE 
workforce members need in terms of specialized knowledge and competencies (National Research Council, 2015). 
As a result, ECE professionals “... need access to high-quality professional learning that supports them in the 
acquisition and application of the competencies they need, both in degree- and certificate-granting programs and 
during ongoing practice throughout their career” (National Research Council, 2015, p. 494). 


Many ECE professionals across the country attain the Child Development Associate (CDA) credential, a nationally 
recognized entry-level ECE credential based on a set of competency standards, which is administered through the 
Council for Professional Recognition. According to the council, more than 370,000 individuals have received the 
CDA credential (Council for Professional Recognition, n.d.), and there is suggestive evidence that children taught 
by providers with the CDA credential experience some positive academic benefits—in, for example, color, number, 
and letter identification, as well as rhyming—compared with children taught by providers without the credential 
(Early et al., 2006). 


Nationwide, the ECE workforce has lower educational attainment than K-12 teachers. Only 46 percent of 
preschool teachers age 25 and older have at least a bachelor’s degree compared with 95 percent of elementary 
and middle school teachers. About 16 percent of preschool teachers have an associate degree, 24 percent have 
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some college experience (with no degree), 13 percent have a high school diploma or equivalent, and 1.6 percent 
have less than a high school diploma (U.S. Department of Labor, 2018). Among ECE workforce members who are 
not preschool teachers, educational attainment is lower still: only 19 percent of nonpreschool ECE workforce 
members have at least a bachelor’s degree, 11 percent have an associate degree, 27 percent have some college 
experience (with no degree), 31 percent have a high school diploma or equivalent, and 13 percent have less than 
a high school diploma (U.S. Department of Labor, 2018). 


Wages for ECE professionals are also low. Nationally, the 2016 median annual wage was $28,790 for preschool 
teachers ($13.84 median hourly wage; U.S. Department of Labor, 2017a) and $21,170 for other ECE workforce 
members ($10.18 median hourly wage; U.S. Department of Labor, 2017b). The median annual wage was $57,160 
for elementary school teachers and $59,170 for secondary school teachers (data on hourly wages were 
unavailable; U.S. Department of Labor, 2017c, 2017d). 


Education and wages of Oregon’s early childhood education workforce 


Oregon’s licensed-care ECE sector employs roughly 25,000 people. About 75 percent of them work in centers 
(including public preschools and Head Start facilities), 13 percent work in large home-based care settings, and 12 
percent work in small home-based care settings. The sector includes infant/toddler care as well as care for 3- and 
4-year-olds. Approximately 30 percent of Oregon’s ECE workforce reported an education level of a high school 
diploma or GED or below (Oregon Center for Career Development in Childhood Care and Education and the 
Oregon Childcare Research Partnership, 2019). About 35 percent of ECE workforce members who provided 
education data to the Oregon Registry Online reported having a bachelor’s degree or higher, while 13 percent had 
an associate degree and 22 percent had some college, a certificate, or a credential (such as the CDA). These figures 
include all licensed early learning professionals (both preschool and child care workers), although 27 percent of 
the licensed ECE workforce did not self-report an education level (Oregon Center for Career Development in 
Childhood Care and the Oregon Childcare Research Partnership, 2019). Center-based staff members tended to 
have higher education levels than other ECE workforce members, while staff members in home-based care 
settings tended to have lower education levels (Portland State University & Oregon State University, 2017). Annual 
mean wages in Oregon were similar to national wages, at $30,230 for preschool teachers and $24,460 for other 
child care workers (data on hourly wages were unavailable; U.S. Department of Labor, 2017e). 


National initiatives to increase the education of the early childhood education workforce 


Multiple initiatives in the United States seek to improve the quality of the ECE workforce and the training its 
members receive through both formal education pathways and professional development opportunities (Gomez 
et al., 2015). While a clear link between higher ECE workforce education and child outcomes has not been 
established—results from prior research are mixed and suffer from study limitations that prohibit causal 
interpretation of the findings (National Research Council, 2015)—state policy leaders hypothesize that increasing 
the education level of the ECE workforce (particularly of workforce members with less than an associate degree) 
will result in higher quality care for children (Manning et al., 2017). They also hypothesize that ongoing 
professional development is useful for ECE professionals at every education level (Manning et al., 2017). Efforts 
to improve training for the ECE workforce often include financial incentives—including scholarships and education 
awards—to encourage participation in formal education or professional development that is linked with career 
pathways (often referred to as a career lattice; Ackerman, 2004). These incentives might help achieve a higher 
quality and more stable workforce (Totenhagen et al., 2016). However, there is little research to inform the 
appropriate design of these incentive approaches, especially regarding what levels or types of incentives are most 
effective at achieving a more highly qualified workforce. 
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Career lattices and workforce registries 

Policymakers and researchers have recognized the need for systems of professional development that incorporate 
career pathways for ECE professionals. As such, career pathways have become a common feature of state 
professional development systems across the country (Karoly, 2012; LeMoine, 2008; Limardo et al., 2016). These 
initiatives to bring professional development systems to ECE workforce members are part of broader efforts to 
professionalize the field given the relatively low beginning education level and specialized knowledge among ECE 
workforce members (Allen & Kelly, 2015). Career pathways outline professional development steps that 
individuals can follow, including key milestones such as credits or degrees earned or certified training hours 
completed. Some pathways include financial incentives tied to the achievement of key milestones. Career 
pathways serve the dual purpose of meeting both the education and professional development needs of the ECE 
workforce while meeting the sector’s needs for an educated and skilled workforce (Limardo et al., 2016). 


ECE workforce members might choose to engage in professional development to increase their knowledge and 
skills in the field, meet licensure or other professional requirements, advance in the profession, earn higher wages, 
or some combination of these factors. Karoly (2012) further highlights the unique need for engagement in 
professional development among ECE workforce members: “... many ECE providers begin caring for children 
before they have had formal professional development, often as classroom assistants or home-based providers” 
(p. 6). 


Career pathways, ladders, or lattices have become a prominent feature in many state ECE systems, with 37 states 


“ 


having some form of career pathway that outlines the trainings and experiences necessary for ECE professionals 
to advance their career in defined ways (Missouri Department of Social Services, 2014). A workforce registry is 
another key component of a robust state professional development system for ECE workforce members, as it 
supports both the monitoring and the evaluation of the professional development system through the tracking of 
ECE workforce members, their education level, credentials, training experience, employment history, and position 
on the career ladder (Karoly, 2012). As of 2018, 42 states and the District of Columbia and Puerto Rico had an ECE 
workforce registry (King et al., 2020; Limardo et al., 2016). However, across all registries, only 69 percent of 
licensed centers had at least one director and one staff member in the registry, and only 44 percent of licensed 
center directors were in the registry (Mayfield, 2017). This indicates a lack of workforce registry participation, 
which could hamper state monitoring and evaluation of the state professional development system. In at least 10 
states, including Oregon, registry sign-up is mandated only for staff members in licensed ECE programs. In other 
states registry sign-up is voluntary regardless of workplace setting (Ackerman, 2016). 


Oregon’s workforce registry and career lattice 


The Oregon Center for Career Development in Childhood Care and Education (OCCD), based at Portland State 
University, manages professional development incentive disbursement, as well as the registry system that 
documents the licenses and professional development of ECE professionals and the career lattice. The registry has 
existed since 1998 and, since 2011, has been managed as an online system called the Oregon Registry Online. All 
workforce members in regulated child care centers and home-based care facilities must be registered to provide 
care to children and must submit documentation of their training and education to the registry. 


OCCD has developed a career lattice that outlines a series of 15 steps representing milestones for those in the ECE 
workforce, such as earning a degree, certificate, or credential (see table B1 in appendix B). There are three tracks 
(degree, credential, or certificate; college course credit; and community-based training), providing multiple 
pathways to move up the lattice. For example, ECE workforce members who are at step 7 have the CDA credential, 
12 college quarter credits or 120 hours of community-based training. To move to step 7.5, they would need to 
earn 8 additional college quarter credits if they have the CDA credential or earn 12 college quarter credits or 80 
additional hours of community-based training if they have 120 hours. The career lattice helps break up the 
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pathway to a higher degree or equivalent training hours into more easily achievable steps that delineate 
cumulative college credits or hours of community-based training needed. For example, the step after earning the 
CDA credential requires earning eight additional college quarter credits, which is the equivalent of two or three 
classes. 


Oregon Registry Online registrants sign up for the career lattice and submit documentation of their education 
level and training hours. This documentation is reviewed by OCCD staff, who then assign a step in the lattice to 
the registrants based on their accumulated education and training. Until April 2020 OCCD offered an incentive 
program based on the step level achieved (see table B1 in appendix B for a description of the career lattice). The 
program was then discontinued due to lack of state funding. 


The incentive program offered $100 on completing 3-9 college quarter credits or 35-90 hours of community- 
based training (steps 3-6); $150 on completion of the CDA credential, 12—40 quarter credits, or 120-400 hours of 
community-based training (steps 7—8.5); and $200 on earning an associate or bachelor’s degree or completing 
600-800 hours of community-based training (steps 9-10). All incentives were paid after OCCD verified step 
completion (for example, after degree receipt) and professional development attainment. Once participants 
received the incentive for a given group of steps, no additional incentives were provided for movement between 
steps within a group. For example, if participants entered the career lattice with the CDA credential, they would 
be assigned to step 7 and, until April 2020, could have received a $150 incentive payment; if they entered with 
300 community-based training hours, they would be assigned to step 8 and, until April 2020, also could have 
received a $150 incentive payment. If participants were at step 7, had already received the $150 incentive, and 
moved to step 7.5, they would not have received an additional incentive. The incentives were not designed to 
compensate directly for hours of time spent gaining education, and the amount of incentive per hour of education 
time decreased as the lattice steps increased. For someone who reached step 7 with 120 hours of community- 
based training, the $150 incentive translated to $1.25 per hour spent in training, or about 12 percent of the 
median hourly wage of $10.72 for ECE workforce members not in a preschool setting (U.S. Department of Labor, 
2017b). 


After entering the lattice (and, until April 2020, potentially receiving an incentive commensurate with their step), 
participants can continue with their education through additional college credits or community-based training 
hours (typically continuing in the same pathway through which they achieved their current step). According to 
OCCD staff members, Oregon Registry Online registrants with a career lattice step often reach the step for the 
CDA credential or associate degree and then do not continue to advance in the career lattice. 


In addition to these monetary incentives for reaching certain steps in the career lattice, OCCD administers a 
scholarship program that pays community college tuition for up to four credits per term (up to approximately 
$450 per term based on 2017/18 tuition rates) for individuals working more than 20 hours a week at a licensed 
care facility. All Oregon Registry Online registrants in the career lattice could apply for the scholarship and submit 
their course registration for reimbursement. Historically, according to OCCD, scholarship uptake has been low. 


However, only about a third of licensed child care workers sign up to have their experience and education 
translated into a step in the career lattice. Two-thirds of the licensed care workforce in Oregon do not sign up for 
the career lattice despite the monetary incentives. Those who do not sign up might be engaging in professional 
development beyond licensure requirements but not reporting it to OCCD, or they might not be engaging in 
professional development. OCCD and the Oregon Department of Education Early Learning Division want to 
encourage participation in the career lattice by providing clear pathways to advancement and to help the state 
understand the education levels of the early childhood workforce, which could in turn guide state policy in this 
area. 
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This study examined ways to provide incentives to and encourage participation in the career lattice and increase 
education levels for those in the career lattice. In addition, because many workforce members stall between steps 
7 and 9 and OCCD is interested in encouraging continued professional development—particularly for workforce 
members with less than an associate degree—the study focused on a narrow band of the lattice, where financial 
incentives might be expected to have a larger impact. This also allowed the study to redistribute the existing 
amount of financial incentive dollars without adding to the cost of the program to the state. 


Related literature on study intervention 


The study team reviewed literature on the implementation and effectiveness of information campaigns and 
financial incentives aimed at increasing continuing education among target populations relevant to the current 
study (for example, students or adults from low-income households or other underrepresented groups, or ECE 
professionals). This research base informed the study’s hypotheses that these types of interventions can improve 
professional development outcomes among ECE professionals. 


While a limited number of studies speak specifically to this study’s hypotheses, the reviewed literature suggests 
the potential for considering both information interventions and financial incentives as a part of the professional 
development system. The limited existing research base also highlights the need for more rigorous research. 


Information interventions using behavioral nudges. Over the last several years the policy community, including 
education practitioners and researchers, has adopted behavioral economics strategies—specifically, the use of 
behavioral nudges—in large-scale social interventions and experiments (Chetty, 2015; Thaler & Sunstein, 2008). 
A behavioral nudge changes the presentation of choices in such a way that people are more likely to choose one 
option than another. The most common behavioral nudges in education experiments are tailored and timely 
information interventions (in the form of mailings, emails, and text messages) aimed at changing an individual's 
behavior in socially optimal ways (National Science and Technology Council, 2015). 


A growing body of literature, focused primarily on postsecondary education access and persistence, provides 
evidence that some information campaigns and information coupled with other supports might be effective at 
encouraging students from low-income backgrounds to apply for need-based aid and to enroll and persist in 
college. Bettinger et al. (2012) found that low-income families who received both an information intervention 
related to college access (a financial aid estimate anchored to tuition costs at nearby institutions) and assistance 
in completing federal financial aid forms were more likely to submit the Free Application for Federal Student Aid 
(FAFSA). The combination of intervention and assistance also increased college enrollment, persistence, and 
receipt of aid. Importantly, individuals who received only the information intervention (without additional tailored 
assistance) did not experience any positive effects on postsecondary outcomes. 


However, in other contexts information-only interventions have been found to positively influence postsecondary 
outcomes. For example, Castleman and Page (2015) found that text message interventions targeted at college- 
intending high school graduates and their parents significantly increased students’ likelihood of enrolling in 
college. Similarly, Hoxby and Turner (2013) found that providing tailored information to high-achieving students 
from low-income households during the college application process encouraged students to apply to and enroll 
in colleges better matched to their academic abilities. Several additional studies have also found positive effects 
of information-only interventions on college enrollment (Barr & Turner, 2018; Castleman & Page, 2017; Hyman, 
2020; Page & Gehlbach, 2017), submission of financial aid application (ideas42, 2015; Page et al., 2020), and 
college persistence (Castleman & Meyer, 2016; Castleman & Page, 2016). 


While the literature to date on the efficacy of behavioral nudge interventions in education has been largely 
positive, a series of recent studies examining the effect of nudge interventions that were scaled up to either the 
state or national level have found no effects on targeted populations (prior interventions were targeted to 
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students graduating from some partner school districts or charter networks or subsets of national or state 
populations—for example, high-achieving low-income students). In what the authors described as “the largest 
FAFSA nudge campaign to date,” Bird et al. (2019, p. 3) found no effects of FAFSA informational nudges on financial 
aid receipt or college enrollment across any groups in a study of about 800,000 students across the nation and 
within one large state. Their study included several experimental conditions, and except for one treatment arm, 
all treatments were information-only nudges that varied by timing, delivery method, content framing, and design. 
This result aligns with other recent studies examining informational nudges administered through a statewide or 
national partner. These studies have found no impacts on college enrollment or persistence (Bergman et al., 2019; 
Gurantz et al., 2019) and no effects on financial aid application submissions (Goldrick-Rab et al., 2019). 


Characteristics of effective information campaigns. The literature on characteristics of effective information 
campaigns is mixed. Hoxby and Turner’s (2013) early nudge study found that families had a strong preference for 
paper materials rather than information shared via email. In addition, given participants’ general suspicion about 
these interventions due to concerns of falling victim to scams, families were more responsive when information 
was accompanied with fee waivers. However, the pilot study from which these findings are drawn took place in 
2009 and 2010, and thus the results might not generalize to today given how quickly preferences and norms 
around technology use can change. Bird et al.’s (2019) recent study of scaling up nudge interventions tested a 
number of variations in the framing of the message content, delivery method (postal mail, emails, and text 
messages), and timing for FAFSA information nudges and found no impacts on financial aid receipt or college 
enrollment for any of the treatment arms. 


Additional literature in behavioral economics provides other examples of the effectiveness of different 
intervention characteristics. For instance, the most effective emails to increase a desired behavior (for example, 
enrollment in savings plans) include a simplified, streamlined message; provide a clear outline of the action steps 
or behaviors required; and emphasize the long-term benefits of the action (for example, saving money; National 
Science and Technology Council, 2015). Other research highlights the importance of providing personalized 
information to study participants. For example, a study examining the effectiveness of information interventions 
targeted toward parents of school-age children found that personalized text messages (for example, including 
student-specific information with actions or suggestions tailored to a student’s skill level) were more effective 
than generic information-only nudges (Doss et al., 2019). Similarly, research on the role of text messages in 
increasing loan repayment found that including the loan officer’s name improved repayment (Karlan et al., 2012). 
Further, when sending nudges to work email addresses, emails scheduled to be received at lunchtime might 
maximize open rates (National Science and Technology Council, 2015). 


Finally, findings on the use of loss-framed versus gain-framed content in behavioral nudges are mixed. Loss- 
framed content focuses on what an individual might lose by not taking up the treatment, while gain-framed 
content focuses on what an individual might gain by taking up the treatment (National Science and Technology 
Council, 2015). The National Science and Technology Council (2015) found that loss-framed emails were more 
effective than gain-framed emails. Yet, Karlan et al. (2012) found no differences in loss versus gain framings in text 
messages. 


Financial incentives in the early childhood education context. The empirical evidence on the effectiveness of 
financial supports to encourage ECE workforce members to continue their education or participate in professional 
development activities has grown over the last two decades but remains sparse. Three literature reviews on the 
extant research suggest that state programs that include financial supports aimed at increasing ECE workforce 
members’ education levels or participation in professional development activities might have been effective in 
achieving those outcomes (Park-Jadotte et al., 2002; Weber & Trauten, 2008; Whitebook & Bellm, 2004). 
However, this literature is largely descriptive and, in many cases, does not employ a comparison group. Moreover, 
the size of the financial incentive offered varied widely across programs. The study team could not locate any 
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experimental evidence on the impact of financial supports on ECE workforce members’ educational attainment 
or professional development outcomes or a clear consensus as to the size of the incentive needed to influence 
behavior. 


Park-Jadotte et al. (2002) concluded that state programs that provided direct financial support or incentives for 
increased education or training, such as bonuses or salary supplements using a variety of compensation strategies 
(stipends, wage ladders, scholarships, and wage supplements), were associated with higher worker education 
levels. Only one evaluation included in the review employed a comparison group (without randomization), while 
the other studies used designs that compared participants pre- and post-intervention. The size of the financial 
incentives in the seven programs in the literature review varied widely. In one program the maximum award at 
the lowest tier was $500, with the stipend increasing to $6,000. In another program individuals who obtained the 
CDA credential were awarded a stipend of $200, moving up to an award of $1,000 for individuals who obtained a 
bachelor’s or master’s degree. Another program rewarded participants with either a one-time bonus in the range 
of $100-S700 or a 4—5 percent pay increase. 


In a 2004 review of the evaluations on California’s financial support initiatives for the child care workforce, the 
Compensation and Recognition Encourage Stability (CARES) program, Whitebook and Bellm (2004) concluded that 
participants in programs that tied financial incentives (in this case, stipends) to professional development often 
exceeded required education and training levels and exhibited strong involvement in professional development. 
Again, the evaluations included in this review did not rely on methods that support causal interpretations of the 
findings; only two of the evaluations included a comparison group (without randomization), and others relied 
largely on designs that compared participants’ pre- and post-program participation. The financial incentives in the 
programs studied ranged from $80 to $200 at the entry level and from $2,000 to $6,000 per year at the top of the 
education ladder (Whitebook & Bellm, 2004). 


Similarly, Weber and Trauten (2008) found that programs tied to compensation initiatives (including scholarships, 
stipends, and annual bonuses) have been associated with advancement on participants’ state career ladders and 
completion of various credentials or other degree programs. Although, as the authors noted, most of these studies 
were correlational and did not support causal conclusions. The studies included in the Weber and Trauten (2008, 
p. 24) review used stipends that ranged from “S100 for the lowest level in the state with the lowest enhancements 
to $5,100 for the highest level in the state with the highest enhancements;” the median lowest stipend was $300, 
and the median highest was $3,000. 


More relevant to the current study, a recent study of an Oregon scholarship program to support the professional 
development of the ECE workforce and the state’s education awards program found a positive association 
between scholarship use and reaching higher steps in the state’s career lattice (Weber & Grobe, 2014). The study 
used statewide administrative data to compare scholarship recipients and nonrecipients among all individuals in 
the child care workforce, so the findings are not causal. The scholarship program provides “financial support to 
reduce barriers to training and education for providers working in home-based childcare, center care, Head Start, 
and before/after school programs,” and the education awards are financial incentives that reward workforce 
members for education achievements tied directly to their progression along the state’s career lattice (Weber & 
Grobe, 2014, p. i). Scholarship award amounts vary from $100 to $2,400, depending on the type of scholarship 
the individual applies for (Weber et al., 2013). The education awards range from $100 to $500. Additional findings 
from Weber and Grobe (2014) suggest that participants who received more than one scholarship participated in 
more training hours, and among participants who received a scholarship, receiving more education awards was 
associated with higher lattice steps. 


Other evidence also supports the link between financial incentives and career progression among ECE 
professionals. For example, a 2003 study on a scholarship program in Wisconsin found that teachers completed, 
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on average, 10 credits more than they were otherwise required to complete, while assistant teachers and other 
ECE professionals completed, on average, 13 more credits than they were otherwise required to in the absence 
of program participation. That report presents descriptive analyses of scholarship recipients’ education and 
income trajectories (and statewide benchmark figures when available), so findings are descriptive rather than 
causal. The scholarships in the Wisconsin study were awarded as bonuses that ranged from $200 to $750, and 
some participants who worked in ECE centers received either bonuses ranging from $150 to $350 from their 
center or a raise of 1—2.5 percent (Adams et al., 2003). 


A 2008 survey of participants in the California CARES program concluded that financial provisions “including 
stipends, gift cards, and educational or safety supplies” provided by the program were more influential in 
participants’ decisions to participate than other support services offered, such as professional development 
activities, trainings, and academic counseling (Harder+Company, 2008, p. 14). The program also increased 
participants’ reported desire to remain in the ECE workforce and supported participants’ career advancement to 
a moderate degree (Harder+Company, 2008). 


Another evaluation found that a Bay Area ECE retention incentive program increased the number of ECE college 
courses and trainings in which staff members participated (Bridges & Carlat, 2003). That report compared 
outcomes between ECE program staff members in counties with incentive programs and ECE program staff 
members in a comparison county and as such suffers from selection bias. The awards in the Bay Area initiative 
ranged from $475 to $5,100 and were awarded based on tenure and education level (Bridges & Carlat, 2003). 


Qualitative evidence from recent evaluations of statewide programs in California and North Carolina provide 
further support for the hypothesis that financial incentives are an important strategy to support the continued 
professionalization of the ECE workforce. In California an evaluation of the statewide quality rating and 
improvement system found that a majority of surveyed staff members who had participated in quality 
improvement efforts (such as professional development) reported that the availability of financial supports were 
at least somewhat important to their decision to participate (Quick et al., 2016). Meanwhile, a 2013 study of North 
Carolina’s compensation program reported that more than 80 percent of those who participated in the scholarship 
program said it “ease[d] the financial burden of education costs” (Child Care Services Association, 2013, as cited 
in National Research Council, 2015, p. 468). 


Relatedly, a study from the behavioral economics literature provides insight on the relationship among financial 
incentives, the behavioral framing of those incentives, and desired outcomes. Specifically, Fryer et al. (2012) found 
that pay-for-performance incentives offered to K—-8 teachers in a participating district and framed as “losses” 
(where teachers were paid a bonus at the beginning of the academic year and told they would have to return 
some portion if their students did not meet academic performance targets) improved teacher performance 
relative to bonuses framed as “gains” (where bonuses were paid at the end of the school year if students achieved 
some level of successes) and to teachers who received no financial incentive. 


Given the low average wages in the ECE workforce, a related body of literature on the role of financial incentives 
in supporting low-income adults in community college provides additional relevant insights. Several studies have 
found an association between providing financial incentives (for example, performance-based scholarships or 
more comprehensive supports that include financial incentives) and higher community college enrollment and 
earned credits among low-income students (Cha & Patel, 2010; Richburg-Hayes et al., 2009; Scrivener & Weiss, 
2009). The dollar amounts of the incentives varied across studies, but the evidence suggests that providing 
financial supports can encourage college-going among low-income populations. Studies with higher dollar 
amounts for incentives would not have been scalable to the state level in Oregon given existing budget allotments 
for the incentive program. 
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Effectiveness of career pathways in increasing educational attainment. Two evaluations of career pathway 
programs aimed at increasing the educational attainment of adults and young adults found positive effects of 
participation on program completion, GED passing rates, and postprogram enrollment in postsecondary education 
(Martin & Broadus, 2013; Miller et al., 2016). Additional research on career pathways for high school students 
suggests positive effects on educational progression and attainment. 


A series of studies on California's Linked Learning initiative, which featured standalone small schools or academies 
within larger high schools in public school districts in California, showed that students participating in Linked 
Learning were more likely to graduate from high school; less likely to drop out; and earned, on average, more 
credits (Warner et al., 2016). The program also had a positive long-term effect on college enrollment among 
students who entered high school with low academic achievement, as well as a positive effect on four-year college 
enrollment for Black students (Caspary & Warner, 2017). Additional studies of career pathways for high school 
students have found positive effects on credit accumulation and progression toward graduation (Castellano et al., 
2011) and on high school graduation and attainment of postsecondary credentials (Berger et al., 2013). 


Effectiveness of career ladders in other settings. A descriptive study on health training pathways in California's 
community colleges found that most students that start a health CTE program obtain an award or credential; 
however, the vast majority of students earn only one award (Bohn et al., 2016). A separate study evaluating the 
labor market returns of participation in California’s community college career pathways found average returns to 
career and technical education certificates and degrees ranging from 12 percent to 23 percent, with the largest 
returns for programs in the healthcare sector (Stevens et al., 2018). 


In a series of studies on nine career pathways being evaluated under the Pathways for Advancing Careers and 
Education study sponsored by the Administration for Children and Families, several impact evaluations have found 
that program participation increases postsecondary enrollment, credit accumulation, and attainment of college 
credentials (Farrell & Martinson, 2017; Gardiner et al., 2017; Rolston et al., 2017). Two of the three programs 
evaluated were focused on pathways to health services careers, and one provided general career pathways 
support toward high-demand occupations, of which nursing and health professions were the most common. 


In sum, extant literature demonstrates a range of financial incentive amounts at different points in time or for 
different education levels and provides evidence that even small financial incentives might help motivate the ECE 
workforce to engage in professional development. Furthermore, financial incentives are often studied in 
combination with other program supports, so the isolated impact of the financial incentive is understudied. 
Behavioral nudges have been shown to work in some other contexts, and the study team hypothesized that they 
might have an impact in this setting (and tested this hypothesis with this study). In addition, nudges are cost 
effective and might be easier to implement across larger populations. 
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Appendix B. Methods 


This appendix provides details on the study data, interventions, key variables and outcome measures, sample, 
methodology, and protocols. 


Data 


This study relied on Oregon Center for Career Development in Childhood Care and Education (OCCD) 
administrative data from the Oregon Registry Online, survey data collected by OCCD, the treatment indicators 
generated from each randomization, and OCCD staff interviews. Each of these data sources is described in more 
detail below. 


Administrative data from the Oregon Center for Career Development in Childhood Care and Education. The 
Regional Educational Laboratory (REL) Northwest entered into a data-sharing agreement with OCCD to access 
information from the Oregon Registry Online on early childhood education (ECE) workforce members. All 
workforce members in licensed ECE facilities? in Oregon must participate in the Oregon Registry Online, which 
documents the licenses and professional development of ECE professionals in the state. For workforce members 
who participate in the career lattice, information is available on education background and credits earned, career 
lattice step, incentives earned or received, work experience, and demographics (race/ethnicity, primary language, 
and birthdate). For workforce members who do not participate in the state’s career lattice, information is available 
on workplace name, workplace address, and license status. 


The study team obtained point-in-time data in summer 2018 to randomly assign Oregon Registry Online 
registrants and career lattice participants to treatment and control groups. Administrative data from September 
2018 (first month of the intervention) and December 2019 (three months after the last nudge) were used for 
demographic information, workplace characteristics, and outcomes in the analysis. Outcome data, such as career 
award and scholarship take-up data, collected through December 2019 were included to account for any delays 
in filing paperwork to enroll in education by early fall 2019. The Oregon Registry Online data are continually 
updated, with workforce members and OCCD staff entering and updating information as it is received. The data 
also include historical workforce information, with start and end dates of positions for workforce members. This 
data source was used to answer research questions 1, 2, and 3 (including subquestions). 


Additionally, the study team obtained OCCD email distribution data provided via its email platform, Emma. These 
data included information for each treatment group for each of the five mailings (September 2018—July 2019) 
about the number of emails sent and received, opened emails, links clicked within emails, and recipient opt-out 
of further emails. This data source was used to answer research question 4a, as well as the treatment-on-the- 
treated analyses for research questions 1, 2, and 3. 


Survey data collected by OCCD. OCCD conducted a survey from July 20 to August 21, 2018, to gather baseline data 
and enroll study participants for both randomizations. OCCD also administered a follow-up survey from 
September 5 to October 28, 2019, that included many of the same questions as the baseline survey to compare 
responses across time. However, the follow-up survey also included questions regarding receipt of behavioral 
nudge and incentive and scholarship information, as well as professional development attainment for participants 
in the incentive randomization. The REL Northwest advised on survey content and ensured that appropriate 
consent processes were followed. All OCCD protocols were approved by the Education Northwest Institutional 
Review Board (IRB) to ensure protection of human subjects. There was a separate survey for each randomization 


1 In addition to ECE services for children up to age 5, licensed ECE facilities may provide care and education services for children up to age 
12, referred to as school-age care. 
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and for baseline and follow-up; all surveys were available in English and Spanish (see protocols at the end of this 
appendix). 


Key measures for the surveys of participants in both randomizations were demographic characteristics, workforce 
participation, and identification as an ECE professional. For the incentive randomization, additional key measures 
included reported motivation to take a college course and perspectives on the information received through the 
behavioral nudge. Additional key outcomes for the incentive randomization surveys included reported enrollment 
in a college course and the number of college courses enrolled in. This data source was used to answer research 
questions 1b, 2b, 3b, and 4b. 


Treatment indicators. These variables, created by the study team as part of the randomization process (see the 
next section for more details), indicated whether the ECE professional was assigned to a control group or one of 
the treatment groups and to which treatment condition the ECE professional was randomly assigned. This data 
source was used to answer research questions 1, 2, 3, and 4. The sample was all available data from 
randomization. 


Interviews with OCCD staff. The study team conducted semi-structured interviews with key OCCD staff members 
in April 2020 to gather information about how the treatment was implemented. Interview data were used to 
answer research question 4. The protocol is included at the end of this appendix and was submitted to Education 
Northwest’s IRB for review and approval. 


Interventions 


The study consisted of two randomized controlled trials (RCTs)—a sign-up randomization and an incentive 
randomization—examining registration for the career lattice (sign-up randomization) and movement up the 
career lattice (incentive randomization). This section describes the three interventions provided to the treatment 
groups in the two RCTs. 


Sign-up randomization. To be in the career lattice, participants needed to sign up to have their education 
background evaluated by OCCD. According to OCCD, about two-thirds of the ECE workforce had not signed up for 
the career lattice as of July 2018. ECE workforce members could learn about the incentive for registering for the 
career lattice from OCCD via several means, including the OCCD website, information distributed at trainings, and 
through their workplace. 


The first RCT (the sign-up randomization) examined how a behavioral nudge (an email inviting and encouraging 
career lattice participation) influenced sign-up for the career lattice. OCCD and the Oregon Department of 
Education Early Learning Division wanted to encourage career lattice participation to help the state track the ECE 
workforce’s education levels, to guide policy in this area, and to connect the workforce to pathways that 
encourage additional education and training. The sign-up randomization consisted of ECE workforce members 
who were registered in the Oregon Registry Online but not in the career lattice. 


OCCD recruited participants through a survey sent out on July 20, 2018, to all eligible workforce members (10,716 
individuals in total). Based on survey responses, the study team randomized participants into a control or a 
treatment group assignment: 


e Control (business as usual): ECE staff members working in licensed ECE settings and registered in the Oregon 
Registry Online (that is, all staff in licensed care settings) did not receive any emails from OCCD encouraging 
them to sign up for the career lattice. These individuals had not signed up for the career lattice as of the time 
of randomization. 


e Treatment: ECE staff members in licensed ECE settings and registered in the Oregon Registry Online (that is, 
all staff in licensed care settings) received email nudges from OCCD encouraging them to sign up for the career 
lattice. The emails also mentioned the existing monetary incentives for reaching certain steps in the career 
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lattice. After randomization of eligible participants who took the initial survey, the treatment group received 
the first email nudge on October 12, 2018, and reminder nudges on December 14, 2018, and February 15, 
May 9, and July 26, 2019 (see figure B1 for an image of the initial nudge). These individuals had not signed up 
for the career lattice as of the time of randomization. 


Both treatment and control groups were invited to complete a follow-up survey on September 5, 2019. 


Figure B1. Sign-up randomization email 


Dear Ashley Pierson, 


Join the Oregon Registry Steps program to be recognized as an early childhood education 
professional and advance your career! 


The Oregon Registry Steps are a sequence of twelve steps based on training and 
education in the childhood care and education field. The Oregon Registry Steps: 


e Recognize your training and education related to childhood care and education. 

e Provide a pathway for your future professional development. 

e Open doors to money related to your Registry Step (Education Awards, Enhanced 
Rate, & Scholarships). 

¢ Connect you to others who care about Oregon's children and families. 


Your education and training may also qualify for an education award of $100, $150, or 


even $200! 
Sign up today at Oregon Registry Steps! 
Sincerely, 


Sarah Myers, Oregon Registry Coordinator 
Oregon Center for Career Development 
503-725-5834 


Note: This image is of the first nudge, sent to the treatment group on October 12, 2018. 
Source: Oregon Center for Career Development in Childhood Care and Education. 
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Incentive randomization. The second RCT (the incentive randomization) involved ECE professionals who had 
already signed up for the career lattice and had an assigned step as of the time of randomization. Specifically, this 
analysis tested whether nudges combined with a different monetary incentive or with automatic scholarship 
enrollment encouraged movement in the career lattice and increased continuing education and professional 
development. To move up a step in the career lattice, workforce members need to submit paperwork to OCCD 
documenting their professional development hours or college coursework. Workforce members might have 
engaged in professional development or taken college courses but failed to submit their paperwork to OCCD and 
thus their step in the lattice might not reflect their true education or training level. To receive the monetary 
incentive for reaching certain steps, workforce members also need to fill out an application for the incentive and 
submit a W-9 form for the payment. 


OCCD recruited participants through a survey sent out on July 20, 2018, to all eligible ECE workforce members. 
After randomization of eligible participants who took the survey, treatment group members received their first 
nudge on August 31, 2018, and reminder nudges on November 16, 2018, and February 15, May 9, and July 26, 
2019 (see figure B2 for an image of the initial nudges for both arms). Both treatment and control groups were 
invited to complete a follow-up survey on September 5, 2019. 


The treatment and control groups for the incentive randomization are described below. Eligible participants in 
each group were those who signed up for the study; were in step 7, 7.5, or 8 of the career lattice at the beginning 
of the study period (all pre-associate degree); and who had already received the $150 incentive for reaching steps 
7-8.5. The study team randomly assigned eligible participants into one of three groups: 


e Control (business as usual): 


o These participants did not receive an email from OCCD with information on incentives, scholarships, or 
encouragement to take courses or professional development hours. 


o These participants were eligible for the business-as-usual OCCD incentive scheme in place until April 2020 
(see table B1) in which payments were provided on completion of a credential, certificate, degree, or 
certain number of professional development hours at specific career lattice steps (incentives provided 
under the current scheme at the first step reached within steps 3-6, steps 7-8.5, and steps 9-10). To 
receive this payment, workforce members needed to fill out an application for the payment and submit 
documentation of the professional development or coursework. 


o These participants could choose to sign up for the business-as-usual scholarship program but were not 
encouraged to do so. If they chose to sign up, they needed to apply and be approved prior to accessing 
scholarship funds. 


e Treatment 1—monetary incentive: 


o These participants received emails from OCCD with information on incentives and encouragement to take 
courses or professional development hours. 


o These participants were eligible for a modified OCCD incentive scheme that rewarded movement toward 
an associate degree. The treatment offered an additional $100 incentive at each of steps 7.5, 8, and 8.5, 
up to a total of $200 to encourage continued movement in the lattice (no additional amount would be 
disbursed at step 9; see table B1). Movement of one step (for example, step 7 to 7.5) would result in a 
$100 incentive payment, while movement of two steps (for example, step 7 to 8, step 7.5 to 8.5, or step 
8 to 9) would result in a payment of $200. These monetary amounts represented a redistribution of the 
available funding per person under the business-as-usual award program (that is, $200 on reaching step 
9 or 10), meaning that this new incentive scheme could be expanded statewide without increasing the 
cost to the state if it proved successful. This group was eligible for the new incentive scheme only and not 
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oO 


for the business-as-usual scheme. To receive the payment, the workforce member needed to fill out an 
application for the payment and submit documentation of the professional development or coursework. 


These participants could sign up for the business-as-usual scholarship program but were not encouraged 
to do so. If they signed up, they needed to apply and be approved prior to accessing scholarship funds. 


e Treatment 2—automatic scholarship enrollment: 


oO 


These participants received emails from OCCD with information that they had been enrolled in a 
scholarship program and could be reimbursed for community college tuition. The emails included 
information about college term registration dates and support available from OCCD to help them decide 
what classes to take. They did not receive information on incentives tied to the career lattice or 
encouragement to pursue professional development. 


These participants were eligible for the business-as-usual incentive scheme. 


These participants were automatically enrolled in a scholarship program that pays for up to six credits of 
community college tuition per term. They did not have to fill out an application and be approved prior to 
accessing scholarship funds; they needed only to send in their course registration to OCCD. For the July 
2019 final reminder email regarding the scholarship enrollment, the email included notice that the 
number of credits paid for was reduced from six to four for the next term. 
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Table B1. Career lattice steps and incentives for the early childhood education workforce in Oregon, 2018 


Incentive 
Degree, credential, College course Community-based ENETi tele) (-mualqoley=4 a) Monetary incentive 
Step certificate track credit track training track April 2020) treatment 
1 Registered with the Office of Child Care, met Department of Human No incentive na 
Services enhanced rate requirements, or 12 hours (treatment incentive 
program applies only 
2 Step 1 plus 20 hours No incentive to steps 7.5, 8, and 
8.5) 
3 3 quarter credits 3 quarter credits or 35 hours $100 upon reaching 
35 hours steps 3-6 (available 
once for the first 
4 5 quarter credits 5 quarter credits or 50 hours eligible step) 
50 hours 
5 7 quarter credits 7 quarter credits or 70 hours 
70 hours 
6 9 quarter credits 9 quarter credits or 90 hours 
90 hours 
7 Child Development 12 quarter credits 120 hours $150 upon reaching 
Associate credential steps 7—-8.5 (available 
fF FTF TTT T0290 IIo _ NCE fr the first 
7.5 Child Development 20 quarter credits 200 hours eligible step) $100 upon reaching 
Associate credential step 7.5, cumulative 
plus 8 quarter credits maximum at $200 
8 Articulated certificate 30 quarter credits 300 hours $100 upon reaching 


step 8, cumulative 
maximum at $200 


8.5 Articulated certificate 40 quarter credits 400 hours $100 upon reaching 
step 8.5, cumulative 
maximum at $200 


9 Associate degree 60 quarter credits 600 hours $200 upon reaching No incentive 
steps 9-10 (available 
9.5 Step 9 plus 10 upper- 70 quarter credits 700 hours once for the first 
division credits eligible step) 
10 Bachelor’s degree 80 quarter credits 800 hours 
11 Master’s degree na na No incentive 
12 Doctorate degree na na No incentive 


na is not applicable. 

Note: Steps indicate cumulative credits or hours. One quarter credit is equivalent to 30-36 hours of time (Portland Community College, 2017). Most steps 
specify that credits and hours be in specific core knowledge categories or that degrees be in certain content areas. Refer to Oregon Center for Career 
Development in Childhood Care and Education (2018) for details. Community-based training refers to professional development offered by OCCD partners 
and trainers. The incentive program was available to workforce members who applied for awards based on their step level through April 2020. 

Source: Authors’ compilation based on information from the OCCD. 
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Figure B2. Incentive randomization email 


Panel A. Monetary incentive 
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Dear Ashley Pierson, 


Due to your participation in the Oregon Registry Steps, you may be eligible* for a special 
education award program where you can earn $100 by moving up to your next step on the 
Oregon Registry Steps! Consider registering for a college course or community training 
event this fall to help you reach your next step. View the Oregon Registry Online Training 
Calendar to see upcoming training events and dates or Oregon Registry and Community 
Colleges to find Oregon community college early childhood programs. 


You could receive $100 for each new step you reach, up to a total of $200. For example, if 
you are currently at Step 7, you could receive $100 when you reach Step 7.5 and another 
$100 when you reach Step 8. See the table below to determine your education award 
eligibility based on step movement. 


Starting Step |Next Step / Award | Next Step / Award 
7 7.5/$100 | 8/$100 
75 | 8/$100 | 8.5/$100 
8 | 8.5/$100 | 9/$100 


Not sure what you need to get to the next Registry Step? Find more information about 
Registry Step requirements at Oregon Registry Steps & Education Awards guide prepared 
exclusively for you. Still have questions? Email us at occd-support-group@pdx.edu. 


Sincerely, 
Sarah Myers, Oregon Registry Coordinator 
Oregon Center for Career Development 


* To be eligible, you must be currently working at an Oregon Office of Child Care licensed 
facility. This special education award program is available until February 2020. Education 
awards will be available for as long as funding allows. The awards are processed in the 
order of the date received by OCCD. Processing may take up to 45 business days. 


Panel B. Automatic scholarship enrollment 


Dear Ashley Pierson, 


Congratulations! You have been enrolled in the Betty Gray Early Childhood Scholarship 
Program through your participation in the Oregon Registry. OCCD is automatically 
enrolling a select group of ECE professionals as part of our professional development 
efforts. This scholarship supports the growth of quality child care in Oregon by providing 
financial support for training and education for childhood care and education 
professionals. 


Your program enrollment is specifically for Oregon community college tuition. The 
scholarship will pay for up to 6 credits per term of Oregon community college tuition for 
classes in the field of early childhood education. 


To receive your scholarship, simply register for class and send our office a copy of your 
proof of registration by September 17 (for PCC the date is September 7). You may attach 
a copy of your class schedule in a reply to this email or mail it to PSU-OCCD, ATTN: 
Scholarship, PO Box 751, Portland OR 97207. If we receive your proof of registration by 
the deadline, we will pay your college directly for up to six credits of tuition (please make 
sure you arrange payment for any additional fees on your student account). Visit Oregon 
Registry and Community Colleges to find an eligible Oregon community college. 


Registration is now open for fall courses, and classes begin on Monday, September 24th. 
If you will need more time to plan for coursework, your scholarship is effective through 
Winter term 2020, as long as you are working in an Oregon Office of Child Care licensed 
facility and scholarship funds are available.We can help you understand what courses to 
take to advance your career. Email us at occd-support-group@pdx.edu with your 
questions. See additional scholarship details at the Questions & Answers prepared 
exclusively for you, including information about other professional development 
scholarship opportunities you may be interested in. 


Sincerely, 
Sarah Myers, Oregon Registry Coordinator 
Oregon Center for Career Development 


Note: These images are of the first nudge, sent to the treatment groups on August 31, 2018. 
Source: Oregon Center for Career Development in Childhood Care and Education. 
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Key variables and outcome measures 


This section describes the key variables and outcome measures used in the study. Several control variables were 
used in regression analyses as covariates for a sensitivity analysis (table B2). 


Table B2. Control variables used in regression analyses 


Incentive 
Incentive randomization— 
randomization— automatic 


Sign-up monetary Xo are) F- Ve al 1°) 
Covariate re-larelolanly4clalela) Makers alahycs) lala) fanteyale Data source 


Three indicators for baseline highest degree 
(some college, associate degree, bachelor's 
degree or higher). High school or less is the 
reference category. 

Two indicators for race/ethnicity (Latinx, or 
other educator of color). White is the 
reference category. 

Indicator measured at baseline for whether 
a participant plans to continue working in 
the field for more than five years. 

Three indicators for the age groups served 
by the program the participant reported 
working for at baseline (infant or toddler 
[only for sign-up], preschool, or school age). 
Three indicators for the type of program a 
participant reported working for at baseline 
(child care center, home-based care, or 
Head Start [only for sign-up]). 

Indicator measured at baseline for whether 
a participant planned to continue formal 
education. 

Indicator measured at baseline for whether 
a participant planned to participate in future 
professional development opportunities. 
Indicator measured at baseline for whether 
an individual had the Child Development 
Associate credential. 

Indicator measured at baseline for an 
individual's identification as a professional. 
Indicator measured at baseline for an 
individual’s motivation 

Two indicators for baseline step (step 7 and 
step 8). 

Two indicators for most recent step path an 
individual took (college course credit and 
community-based training). 

Number of years working in the field at 
baseline. 

Indicator for whether an individual expects a 
raise with a higher degree measured at 
baseline. 

Indicator for whether an individual was 
enrolled in a college course at baseline. 
Indicator for whether an individual reported 
that the workplace was very supporting of 
participating in ongoing professional 
development at baseline. 


Source: Authors’ compilation. 


Baseline survey; Oregon 
Registry Online 
administrative data 
Baseline survey; Oregon 
Registry Online 


administrative data 
Baseline survey 


Baseline survey 


Baseline survey 


Baseline survey 


Baseline survey 


Baseline survey 


Baseline survey 
Baseline survey 
Oregon Registry Online 
administrative data 
Oregon Registry Online 
administrative data 


Baseline survey 


Baseline survey 


Baseline survey 


Baseline survey 
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The study team determined outcome measures based on a logic model of the study interventions and what they 
might be expected to influence in the short, medium, and long term (figure B3). For research question 1 (sign-up 
randomization), the study team constructed an indicator variable for whether an individual enrolled in the career 
lattice as of January 1, 2020, and used an indicator of whether an individual was retained at the same workplace 
during the study period. For research questions 2 and 3 (incentive randomization), the outcome measures 
included an indicator for whether an individual increased a step during the study period, total number of 
community-based training hours recorded, total number of college credit hours earned, and an indicator for 
whether an individual received a scholarship during the study period (for the automatic scholarship enrollment 
group of the incentive randomization only), as well as the indicator of whether the individual was retained at the 
same workplace during the study period. 


The study team also examined additional secondary outcomes that were used in exploratory analysis (questions 
1b, 2b, and 3b) because they are not the main outcomes of interest and they draw from survey data that 
experienced significant attrition. These outcomes included a dichotomous variable denoting an increase in 
identification as an early learning professional at the follow-up survey compared with the baseline survey. For 
example, respondents who responded “somewhat agree” at baseline and “strongly agree” at follow-up when 
asked to what extent they agreed with the statement “I feel like | am an early learning professional” were denoted 
as having an increase. The variable for increase in identification as a professional came from item 20 in the baseline 
survey and item 5 in the follow-up survey (sign-up randomization) and item 14 in the baseline survey and item 3 
in the follow-up survey (incentive randomization). The question was on a 5-point scale from strongly disagree to 
strongly agree, and any increase from strongly disagree to disagree, from disagree to neither agree nor disagree, 
from neither agree nor disagree to agree, or from agree to strongly agree (or increases of more than one level) 
was coded as an increase. Identification as a professional is a key concept in the ECE workforce; increasing 
workforce members’ sense of identification as a professional could contribute to increases in the quality of care 
they provide (National Research Council, 2015). Other aspects of professionalism, such as earning appropriate 
compensation, are not captured in this study’s measure. 


Other secondary outcomes included motivation to take college courses in the next year or the next five years and 
motivation to earn a degree in five years. These variables were constructed from the follow-up survey questions 
and indicate whether respondents’ level of motivation to take a college course in the next year, take a college 
course in the next five years, or earn a degree was high or very high. These motivation indicators were taken from 
items 4, 5, and 6 in the follow-up survey (used in incentive randomization only). The study team and the OCCD 
partners selected these motivation outcomes as potential leading indicators of behavior change (enrolling in and 
completing professional development; see figure B3). 
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Figure B3. Logic model of study interventions 


Inputs Short-term outcomes Medium-term outcomes Long-term outcomes 
(less than 1 year) (1-2 years) (3 or more years) 
* People sign up for career lattice(career * People stayin field as theyfee * People earn degrees and 
Emails encouraging lattice sign-up) more supporteftetention) credential §PD attainment) 
career lattice sign-up * People feel more professional as they are * People take PD to move up a 
more connected with OCCD (identification as ste p(PD attainment) 


a professional) 


a 


e 
Ss 


OCCD is Oregon Center for Career Development in Childhood Care and Education. PD is professional development (encompassing college coursework and 
training hours). 
Source: Authors’ compilation. 


Sample 

This section details the study sample. For the sign-up randomization, OCCD invited all Oregon ECE workforce 
members who were not in the career lattice at the time of randomization to participate (see panel A in figure B4 
for a diagram of eligibility and participation). For the incentive randomization (both the monetary incentive and 
the automatic scholarship enrollment treatment groups), OCCD invited all Oregon ECE workforce members who 
were in the career lattice and were at career step 7, 7.5, or 8 to participate (see panel B in figure B4 for a diagram 
of eligibility and participation). The study team conducted the random assignments in August 2018 for the 
incentive randomization and in October 2018 for the sign-up randomization, using rerandomization (Mihaly et al., 
forthcoming; Morgan & Rubin, 2015) to ensure that the treatment and control groups were similar on important 
covariates. Rerandomization involves conducting repeated randomizations until the difference in characteristics 
between the two groups is small enough to be acceptable (based on a threshold set before the randomization 
starts). The process involves randomizing assignments, checking for balance in characteristics, and discarding the 
assignment until a randomization yields the desired balance. For this study the threshold for an acceptable 
difference was set so that 5 percent of the random assignments would be acceptable. By ensuring that the two 
groups are similar, rerandomization provides more precise and trustworthy estimates of treatment effects 
(Morgan & Rubin, 2015). 
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Figure B4. Consolidated standards of reporting trials (CONSORT) diagram 


Panel A. Sign-up randomization 
Assessed for eligibility 
N= 13,825 
—<_ er 
« No valid email address in ORO n = 3,109 
Invited participants 
n=10,716 


Did not meet inclusion criteria 

¢ Did not complete baseline survey n = 10,166 

¢ Did not consent to participate n = 37 

¢ Did not provide email address n = 75 

¢ Already registered for steps n = 83 

¢ Not employed in ECE n=1 

¢ Opted out of receiving emails from OCCD n=6 


Allocated to treatment group 
n=174 


Lost to follow-up 
n=0 
Discontinued intervention (opted out of emails) 
n=7 


Analyzed 
n=140 
Excluded from analysis 
« Determined to be ineligible because enrolled 
in career lattice by time of study start n = 33 
« Determined to be ineligible because incorrect 
employment record in system n= 1 


Analyzed 
n=149 


Excluded from analysis 
« Determined to be ineligible because enrolled 
in career lattice by time of study start n = 25 


Panel B. Incentive randomization 


Assessed for eligibility 
N=2.874 


Did not meet inclusion criteria 
« No valid email address in ORO n= 148 


Invited participants 
n=2,726 


Completed sign-up trial baseline 
survey but met incentive trial 


inclusion criteria 


¢ Included from sign-up n = 15 


Allocated to monetary incentive 
treatment group 
n=81 


Lost to follow-up 


n=0 


Discontinued intervention 
(opted out of emails) 
n=1 


Analyzed 
n=T77 
Excluded from analysis 


¢ Determined to be ineligible 
because at step 8.5 or higher 
by time of study start n = 4 


Did not meet inclusion criteria 

« Did not complete baseline survey n = 2,438 
+ Did not consent to participate n = 14 

« Did not provide email address n = 42 
«Not at steps 7,75 0r8n=1 

« Did not match in ORO n=2 


Randomized 
n=244 


Allocated to scholarship 
treatment group 
n=81 


Lost to follow-up 
n=0 


Discontinued intervention 
(opted out of emails) 
n=2 


Analyzed 
n=80 

Excluded from analysis 

¢ Determined to be ineligible 


by time of study start n =1 


because at step 8.5 or higher 


Allocated to control group 
n=82 


Lost to follow-up 
n=0 


Analyzed 
n=78 
Excluded from analysis 
¢ Determined to be ineligible 
because at step 8.5 or higher 
by time of study start n = 4 


ECE is early childhood education. OCCD is Oregon Center for Career Development in Childhood Care and Education. ORO is Oregon Registry Online. 
Source: Authors’ compilation. 


Sample characteristics and power calculations. For the sign-up randomization 10,716 individuals were invited to 
participate in the study. Of those invited, 348 agreed to participate (see table B3 and table B6 later in the appendix 
for sample characteristics and outcomes). For the incentive randomization 2,726 individuals were invited to 
participate in the study. Of those invited, 244 agreed to participate (see tables B4, B5, B7, and B8 for sample 
characteristics and outcomes). Table B9 compares characteristics of the study sample to the state sample. Table 
B10 compares characteristics of the study sample for the sign-up and incentive randomizations to those of 
individuals in the workforce but not in the career lattice and to those of individuals at step 7, 7.5, or 8. Table B11 
shows power calculations for these sample sizes. 
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Table B3. Sign-up randomization: Participant characteristics at time of assignment and end of study period, 2018 and 2019 


At time of assignment At end of study period 
Treatment (n = 140) Control (n = 149) Treatment (n = 140) Control (n = 149) Significance 
At time of PNaTale Ke) j 

Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 
Female 94.4 89 92.8 97 94.4 89 92.8 97 ns ns 
Gender missing 36.4 140 34.9 149 36.4 140 34.9 149 ns ns 
Race/ethnicity 

White 65.3 124 62.8 129 65.3 124 62.8 129 ns ns 

Latinx 20.2 124 24.8 129 20.2 124 24.8 129 ns ns 

Non-Latinx educator of color 14.5 124 12.4 129 14.5 124 12.4 129 ns ns 

Race/ethnicity missing 11.4 140 13.4 149 11.4 140 13.4 149 ns ns 
Age (years) 37.3 140 35.0 149 38.7 140 36.4 149 ns ns 
Lives in urban area 86.4 140 77.9 149 86.4 140 77.9 149 ns ns 
Lives in rural area 13.6 140 22.1 149 13.6 140 22.1 149 ns ns 
Primary language 

English 83.7 92 85.1 101 83.7 92 85.1 101 ns ns 

Non-English 16.3 92 14.8 101 16.3 92 14.8 101 ns ns 

Primary language missing 34.3 140 32.2 149 34.3 140 32.2 149 ns ns 
Highest level of education (self-reported) 

High school diploma or less 28.4 134 24.6 138 33.0 103 30.3 109 ns ns 

Some college or other professional 22.4 134 25.4 138 15.5 103 15.6 109 ns ns 

certificate 

Associate degree 10.5 134 11.6 138 13.6 103 15.6 109 ns ns 

Bachelor’s degree or more 38.8 134 38.4 138 37.9 103 38.5 109 ns ns 


Self-reported education missing 4.3 140 7.4 149 26.4 140 26.9 149 ns ns 


At time of assignment At end of study period 


Treatment (n = 140) Control (n = 149) Treatment (n = 140) Control (n = 149) Significance 


At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 


Job characteristics 
Tenure, overall (years) 6.5 106 4.8 111 na na na na ns na 
Tenure missing 24.3 140 25.5 149 na na na na ns na 


Workplace role 


Director/owner 8.5 118 7.3 124 9.3 54 12.7 55 ns ns 
Lead teacher 14.4 118 12.9 124 22.2 54 10.9 55 ns ns 
Teacher 16.1 118 21.0 124 14.8 54 25.5 55 ns ns 
Aide/assistant teacher 44.1 118 48.4 124 27.8 54 30.9 55 ns ns 
Other role 16.9 118 10.5 124 25.9 54 20.0 55 ns ns 
Workplace role missing 15.7 140 16.8 149 61.4 140 63.1 149 ns ns 
Workplace type 

Child care center 53.4 118 53.2 124 na na na na ns na 
Head Start/Early Head Start 16.9 118 21.0 124 na na na na ns na 
Home-based care 14.4 118 10.5 124 na na na na ns na 
Relief nursery, elementary school, or 15.3 118 15.3 124 na na na na ns na 
other 

Program type missing 15.7 140 16.8 149 na na na na ns na 


Rerandomization balance variables 


Plan to continue working in this field for 53.4 118 56.1 123 na na na na ns na 
more than five years 

Plan to continue working in this field for 15.7 140 17.5 149 na na na na ns na 
more than five years missing 

Ages of children in your program: 63.6 118 63.4 123 na na na na ns na 
Infant or toddler 

Ages of children in your program: 78.8 118 82.1 123 na na na na ns na 
Preschool 

Ages of children in your program: 44.9 118 39.0 123 na na na na ns na 


School-age 


At time of assignment At end of study period 


Treatment (n = 140) Control (n = 149) Treatment (n = 140) Control (n = 149) Significance 


At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 

Ages of children in your program missing 15.7 140 17.5 149 na na na na ns na 
Plan to participate in future training 59.2 120 58.4 125 na na na na ns na 
Plan to participate in future training 14.3 140 16.1 149 na na na na ns na 
missing 

Plan to continue formal education 57.5 120 59.7 124 na na na na ns na 
Plan to continue formal education 14.3 140 16.8 149 na na na na ns na 
missing 

Have the CDA credential at baseline 10.4 125 13.1 130 na na na na ns na 
Have the CDA credential at baseline 10.7 140 12.7 149 na na na na ns na 
missing 

Identify as a professional: strongly agree 42.7 117 42.7 124 na na na na ns na 
Identify as a professional: missing 16.4 140 16.8 149 na na na na ns na 
What motivates you to continue 30.7 140 32.2 149 na na na na ns na 


education: missing at baseline 


CDA is Child Development Associate. na is not applicable. ns is not significant. 
Note: No differences between groups for the sign-up randomization were significant at p < .05. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B4. Incentive randomization: Characteristics of monetary incentive participants at time of assignment and end of study period, 2018 and 2019 


At time of assignment At end of study period 


Treatment (n = 77) Control (n = 78) Treatment (n = 77) Control (n = 78) Significance 


At time of At end of 


Participant characteristic Percent Number Percent Number Percent Number Percent Number | assignment study period 
Female 97.4 77 96.1 78 97.4 77 96.1 78 ns ns 
Gender missing 0.0 77 0.0 78 0.0 77 0.0 78 ns ns 
Race/ethnicity 

White 61.0 77 68.0 75 61.0 77 68.0 75 ns ns 

Latinx 23.4 77 24.0 75 23.4 77 24.0 75 ns ns 

Non-Latinx educator of color 15.6 77 8.0 75 15.6 77 8.0 75 ns ns 

Race/ethnicity missing 0.0 77 3.8 78 0.0 77 3.8 78 ns ns 
Age (years) 43.5 77 43.8 78 44.8 77 45.1 78 ns ns 
Lives in urban area 71.4 77 76.9 78 71.4 77 76.9 78 ns ns 
Lives in rural area 28.6 77 23.1 78 28.6 77 23.1 78 ns ns 
Primary language 

English 80.5 77 84.6 78 80.5 77 84.6 78 ns ns 

Non-English 19.5 77 15.4 78 19.5 77 15.4 78 ns ns 

Primary language missing 0.0 77 0.0 78 0.0 77 0.0 78 ns ns 
Highest level of education (self-reported) 

High school diploma or less 22.1 77 16.0 75 27.6 76 27.5 69 ns ns 

Some college or other professional 46.8 77 48.0 75 44.7 76 40.6 69 ns ns 

certificate 

Associate degree 13.0 77 10.7 75 9.2 76 10.1 69 ns ns 

Bachelor's degree or more 18.2 77 25.3 75 18.4 76 21.7 69 ns ns 


Self-reported education missing 0.0 77 3.8 78 1.3 77 11.5 78 . oi 


At time of assignment At end of study period 


Treatment (n = 77) Control (n = 78) Treatment (n = 77) Control (n = 78) Significance 


At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number | assignment study period 


Most recent step number 


Step 7 68.8 77 61.5 78 57.1 77 50.0 78 ns ns 
Step 7.5 16.9 77 15.4 78 15.6 77 12.8 78 ns ns 
Step 8 14.3 77 23.1 78 16.9 77 24.4 78 ns ns 
Step 8.5 0.0 77 0.0 78 2.6 77 10.3 78 ns ns 
Step 9 0.0 77 0.0 78 6.5 77 1.3 78 ns ns 
Step 9.5 0.0 77 0.0 78 0.0 77 0.0 78 ns ns 
Step 10 0.0 77 0.0 78 1.3 77 13 78 ns ns 
Step 11 0.0 77 0.0 78 0.0 77 0.0 78 ns ns 
Most recent step path 
Degree, certificate, credential 20.8 77 16.7 78 22.1 77 17.9 78 ns ns 
College course credit 39.0 77 50.0 78 35.1 77 47.4 78 ns ns 
Community-based training 26.0 77 17.9 78 22.1 77 17.9 78 ns ns 
Combination 14.3 77 15.4 78 20.8 77 16.7 78 ns ns 
Step 1-6 undifferentiated in data 0.0 77 0.0 78 0.0 77 0.0 78 ns ns 
Job characteristics 
Tenure, overall (years) 11.3 68 12.2 68 na na na na ns na 
Tenure missing 11.7 77 12.8 78 na na na na ns na 


Workplace role 


Director/owner 38.9 72 34.3 73 41.2 51 46.0 37 ns ns 
Lead teacher 19.4 72 23.3 73 17.6 51 27.0 37 ns ns 
Teacher 13.9 72 11.0 73 11.8 51 5.4 37 ns ns 
Aide/assistant teacher 22.2 72 23.3 73 21.6 51 13.5 37 ns ns 
Other role 5.6 72 8.2 73 7.8 51 8.1 37 ns ns 


Workplace role missing 6.5 77 6.4 78 33.8 77 52.6 78 ns 


At time of assignment At end of study period 


Treatment (n = 77) (Goya hige) (7a As)) Treatment (n = 77) Control (n = 78) Significance 


At time of At end of 


Participant characteristic Percent Number Percent Number Percent Number Percent Number | assignment study period 
Workplace type 
Child care center 40.0 70 40.3 72 na na na na ns na 
Head Start/Early Head Start 21.4 70 16.7 72 na na na na ns na 
Home-based care 31.4 70 36.1 72 na na na na ns na 
Relief nursery, elementary school, or 7.1 70 6.9 72 na na na na ns na 
other 
Program type missing 9.1 77 7.7 78 na na na na ns na 


Rerandomization balance variables 


Step 7 70.1 77 62.8 78 na na na na ns na 
Step 8 14.3 77 21.8 78 na na na na ns na 
Step missing 0.0 77 0.0 78 na na na na ns na 
Step path: College course credit 41.1 73 51.4 70 na na na na ns na 
Step path: Community-based training 37.0 73 31.4 70 na na na na ns na 
Path missing 5.2 77 10.3 78 na na na na ns na 
Ages of children in your program: 90.1 71 89.0 73 na na na na ns na 
preschool 

Ages of children in your program: 38.0 71 43.8 73 na na na na ns na 
school-age 

Ages of children in your program missing 7.8 77 6.4 78 na na na na ns na 
Tenure, overall (standardized) 32.9 68 43.7 68 na na na na ns na 
Tenure missing 11.7 77 12.8 78 na na na na ns na 
Plan to continue working in this field for 73.6 72 80.8 73 na na na na ns na 


more than five years 


Plan to continue working in this field for 6.5 77 6.4 78 na na na na ns na 
more than five years missing 


Expect a raise with a higher degree at 53.7 67 55.4 65 na na na na ns na 
baseline 


At time of assignment At end of study period 


Treatment (n = 77) Control (n = 78) Treatment (n = 77) Control (n = 78) Significance 
At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number | assignment study period 

Expect a raise with a higher degree 13.0 77 16.7 78 na na na na ns na 
missing 

Enrolled in a college course at baseline 15.9 63 20.6 63 na na na na ns na 
Enrolled in a college course missing 18.2 77 19.2 78 na na na na ns na 
Plan to continue formal education 57.3 68 62.0 71 na na na na ns na 
Plan to continue formal education 11.7 77 9.0 78 na na na na ns na 
missing 

Workplace is very supportive of 55.6 63 50.0 62 na na na na ns na 


participating in ongoing professional 
development at baseline 


Workplace is very supportive of 18.2 77 20.5 78 na na na na ns na 
participating in ongoing professional 
development missing 


* Significant at p < .05; ** significant at p < .01. 

na is not applicable. ns is not significant. 

Note: No differences between groups for the monetary incentive randomization were significant at p < .001. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B5. Incentive randomization: Characteristics of automatic scholarship enrollment participants at time of assignment and end of study period, 
2018 and 2019 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 


At time of Nemelale me) i 


Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 
Female 98.8 80 96.1 78 98.8 80 96.1 78 ns ns 
Gender missing 0.0 80 0.0 78 0.0 80 0.0 78 ns ns 
Race/ethnicity 

White 64.1 78 68.0 75 64.1 78 68.0 75 ns ns 

Latinx 29.5 78 24.0 75 29.5 78 24.0 75 ns ns 

Non-Latinx educator of color 6.4 78 8.0 75 6.4 78 8.0 75 ns ns 

Race/ethnicity missing 2.5 80 3.8 78 2.5 80 3.8 78 ns ns 
Age (years) 40.7 80 43.8 78 42.1 80 45.1 78 ns ns 
Lives in urban area 71.2 80 76.9 78 71.2 80 76.9 78 ns ns 
Lives in rural area 28.7 80 23.1 78 28.7 80 23.1 78 ns ns 
Primary language 

English 80.0 80 84.6 78 80.0 80 84.6 78 ns ns 

Non-English 20.0 80 15.4 78 20.0 80 15.4 78 ns ns 

Primary language missing 0.0 80 0.0 78 0.0 80 0.0 78 ns ns 
Highest level of education (self-reported) 

High school diploma or less 17.5 80 16.0 75 25.7 74 27.5 69 ns ns 

Some college or other professional 45.0 80 48.0 75 35.1 74 40.6 69 ns ns 

certificate 

Associate degree 15.0 80 10.7 75 14.9 74 10.1 69 ns ns 

Bachelor's degree or more 22.5 80 25.3 75 24.3 74 21.7 69 ns ns 


Self-reported education missing 0.0 80 3.8 78 75 80 11.5 78 * ns 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 


At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 


Most recent step number 


Step 7 68.8 80 61.5 78 60.0 80 50.0 78 ns ns 
Step 7.5 11.2 80 15.4 78 13.8 80 12.8 78 ns ns 
Step 8 20.0 80 23.1 78 21.3 80 24.4 78 ns ns 
Step 8.5 0.0 80 0.0 78 1.3 80 10.3 78 ns * 
Step 9 0.0 80 0.0 78 2.5 80 1.3 78 ns ns 
Step 9.5 0.0 80 0.0 78 0.0 80 0.0 78 ns ns 
Step 10 0.0 80 0.0 78 1.3 80 1.3 78 ns ns 
Step 11 0.0 80 0.0 78 0.0 80 0.0 78 ns ns 
Most recent step path 
Degree, certificate, credential 16.2 80 16.7 78 16.2 80 17.9 78 ns ns 
College course credit 55.0 80 50.0 78 53.8 80 47.4 78 ns ns 
Community-based training 17.5 80 17.9 78 16.2 80 17.9 78 ns ns 
Combination 11.2 80 15.4 78 13.8 80 16.7 78 ns ns 
Step 1-6 undifferentiated in data 0.0 80 0.0 78 0.0 80 0.0 78 ns ns 
Job characteristics 
Tenure, overall (years) 10.5 64 12.2 68 na na na na ns na 
Tenure missing 20.0 80 12.8 78 na na na na ns na 


Workplace role 


Director/owner 35.1 74 34.3 73 39.6 48 46.0 37 ns ns 
Lead teacher 24.3 74 23.3 73 27.1 48 27.0 37 ns ns 
Teacher 20.3 74 11.0 73 12.5 48 5.4 37 ns ns 
Aide/assistant teacher 16.2 74 23.3 73 6.3 48 13.5 37 ns ns 
Other role 4.1 74 8.2 73 14.6 48 8.1 37 ns ns 


Workplace role missing 7.5 80 6.4 78 40.0 80 52.6 78 ns ns 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 


At time of NamJale ei 


Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 
Workplace type 
Child care center 44.6 74 40.3 72 na na na na ns na 
Head Start/Early Head Start 20.3 74 16.7 72 na na na na ns na 
Home-based care 31.1 74 36.1 72 na na na na ns na 
Relief nursery, elementary school, or 4.1 74 6.9 72 na na na na ns na 
other 
Program type missing 7.5 80 7.7 78 na na na na ns na 


Rerandomization balance variables 


Step 7 70.0 80 62.8 78 na na na na ns na 
Step 8 18.8 80 21.8 78 na na na na ns na 
Step missing 0.0 80 0.0 78 na na na na ns na 
Step path: college course credit 55.8 77 51.4 70 na na na na ns na 
Step path: community-based training 26.0 77 31.4 70 na na na na ns na 
Path missing 3.8 80 10.3 78 na na na na ns na 
Ages of children in your program: 94.6 74 89.0 73 na na na na ns na 
preschool 

Ages of children in your program: 47.3 74 43.8 73 na na na na ns na 
school-age 

Ages of children in your program missing 7.5 80 6.4 78 na na na na ns na 
Tenure, overall 23.7 64 43.7 68 na na na na ns na 
Tenure missing 20.0 80 12.8 78 na na na na ns na 
Plan to continue working in this field for 83.6 73 80.8 73 na na na na ns na 
more than five years 

Plan to continue working in this field for 8.7 80 6.4 78 na na na na ns na 
more than five years missing 

Expect a raise with a higher degree at 53.5 71 55.4 65 na na na na ns na 
baseline 

Expect a raise with a higher degree 11.2 80 16.7 78 na na na na ns na 


missing 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 
At time of At end of 
Participant characteristic Percent Number Percent Number Percent Number Percent Number assignment study period 

Enrolled in a college course at baseline 25.0 64 20.6 63 na na na na ns na 
Enrolled in a college course missing 20.0 80 19.2 78 na na na na ns na 
Plan to continue formal education 66.2 68 62.0 71 na na na na ns na 
Plan to continue formal education 15.0 80 9.0 78 na na na na ns na 
missing 

Workplace is very supportive of 47.7 65 50.0 62 na na na na ns na 


participating in ongoing professional 
development at baseline 


Workplace is very supportive of 18.8 80 20.5 78 na na na na ns na 
participating in ongoing professional 
development missing 


* Significant at p < .05. 

na is not applicable. ns is not significant. 

Note: No differences between groups for the automatic scholarship enrollment randomization were significant at p < .01. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B6. Sign-up randomization: Confirmatory and secondary outcomes at time of assignment and end of study period, 2018 and 2019 


At time of assignment At end of study period 


Treatment (n = 140) Control (n = 149) Treatment (n = 140) Control (n = 149) Significance 


At time of At end of 
Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 


Confirmatory outcomes 


Enrolled in career lattice 0.0 140 0.0 149 18.6 140 15.4 149 ns ns 
Workplace retention na na na na 80.3 137 81.1 143 na ns 
Secondary outcomes 

Entry step 

Step 1 0.0 140 0.0 149 0.7 140 0.0 149 ns ns 
Step 2 0.0 140 0.0 149 3.6 140 6.0 149 ns ns 
Step 3 0.0 140 0.0 149 5.7 140 2.0 149 ns ns 
Step 4 0.0 140 0.0 149 0.0 140 2.7 149 ns ns 
Step 5 0.0 140 0.0 149 24 140 1.3 149 ns ns 
Step 6 0.0 140 0.0 149 0.0 140 0.0 149 ns ns 
Step 7 0.0 140 0.0 149 2.1 140 4.0 149 ns ns 
Step 7.5 0.0 140 0.0 149 0.0 140 0.7 149 ns ns 
Step 8 0.0 140 0.0 149 0.7 140 0.0 149 ns ns 
Step 8.5 0.0 140 0.0 149 0.0 140 0.7 149 ns ns 
Step 9 0.0 140 0.0 149 0.7 140 1.3 149 ns ns 
Step 9.5 0.0 140 0.0 149 0.0 140 0.0 149 ns ns 
Step 10 0.0 140 0.0 149 2.1 140 3.4 149 ns ns 
Step 11 0.0 140 0.0 149 0.7 140 0.7 149 ns ns 

Highest degree verified in the Oregon Registry Online 

No degree verified 91.4 140 90.6 149 90.7 140 88.6 149 ns ns 
Associate degree 1.4 140 2.7 149 2.1 140 3.4 149 ns ns 
Bachelor’s degree 5.7 140 6.0 149 5.7 140 7.4 149 ns ns 
Master’s degree or higher 1.4 140 0.7 149 1.4 140 0.7 149 ns ns 


Credential verified 0.0 140 0.7 149 1.4 140 2.7 149 ns ns 


At time of assignment At end of study period 


Treatment (n = 140) Control (n = 149) Treatment (n = 140) Control (n = 149) Significance 
At time of At end of 
Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 
Increase in identification as a professional na na na na 18.4 49 28.0 50 na ns 
Increase in identification as a professional na na na na 65.0 140 66.4 149 na ns 
missing 


na is not applicable. ns is not significant. 
Note: No differences between groups for the confirmatory and secondary outcomes in the sign-up randomization were significant at p < .05. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B7. Incentive randomization: Monetary incentive confirmatory and secondary outcomes at time of assignment and end of study period, 2018 
and 2019 


At time of assignment At end of study period 


Treatment (n = 77) Control (n = 78) Treatment (n = 77) (oy nige) (7 As) Significance 


At time of At end of 
Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 


Confirmatory outcomes 


Increased at least one step in the career 83.1 77 82.1 78 19.5 77 17.9 78 ns ns 
lattice 

College credit hours earned between na na na na 15.1 77 19.0 78 na ns 
9/1/2018 and 12/31/2019 

Total community-based training hours na na na na 42.5 77 32.2 78 na ns 
recorded between 9/1/2018 and 

12/31/2019 

Workplace retention na na na na 81.3 75 85.5 76 na ns 
Secondary outcomes 

Received at least one education award na na na na 15.6 77 1.3 78 na am 
Took college course since August 2018 na na na na 19.2 52 30.8 39 na ns 
Took college course missing na na na na 32.5 77 50.0 78 ns ba 
Plan to participate in future professional 82.1 67 73.6 72 80.4 51 81.6 38 ns ns 


development 


Plan to participate in future professional 13.0 77 7.7 78 33.8 77 51.3 78 ns . 
development missing 


Motivation to take a college course in the 37.5 64 39.1 64 48.1 52 30.8 39 ns ns 
next year high or very high 


Motivation to take a college course in the 16.9 77 17.9 78 32.5 77 50.0 78 ns as 
next year missing 


Motivation to take a college course in the 48.4 64 44.4 63 44.2 52 41.0 39 ns ns 
next five years high or very high 


Motivation to take a college course in the 16.9 77 19.2 78 32.5 77 50.0 78 ns i 
next five years missing 


Motivation to earn a degree in the next five 43.8 64 56.3 64 40.4 52 43.6 39 ns ns 
years high or very high 


At time of assignment At end of study period 


Treatment (n = 77) Control (n = 78) Treatment (n = 77) Control (n = 78) Significance 
At time of At end of 

Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 
Motivation to earn a degree in the next five 16.9 77 17.9 78 32.5 77 50.0 78 ns . 
years missing 
Increase in identification as a professional na na na na 18.4 49 16.2 37 na ns 
Increase in identification as a professional na na na na 36.4 77 52.6 78 na . 
missing 


* Significant at p < .05; ** significant at p < .01. 

na is not applicable. ns is not significant. 

Note: No differences between groups for the confirmatory and secondary outcomes in the monetary incentive randomization were significant at p < .001. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B8. Incentive randomization: Automatic scholarship enrollment confirmatory and secondary outcomes at time of assignment and end of study 
period, 2018 and 2019 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 


At time of Natalee) 
Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 


Confirmatory outcomes 


Received a community college scholarship na na na na 6.3 80 2.6 78 na ns 
Increased at least one step in the career 81.3 80 82.1 78 10.0 80 17.9 78 ns ns 
lattice 

College credit hours earned between na na na na 16.0 80 19.0 78 na ns 
9/1/2018 and 12/31/2019 

Workplace retention na na na na 83.7 80 85.5 76 na ns 
Secondary outcomes 

Total community-based training hours na na na na 37.9 80 32.2 78 na ns 
recorded between 9/1/2018 and 

12/31/2019 

Took college course since August 2018 na na na na 35.4 48 30.8 39 na ns 
Took college course missing na na na na 40.0 80 50.0 78 na ns 
Plan to participate in future professional 73.5 68 73.6 72 68.8 48 81.6 38 ns ns 


development 


Plan to participate in future professional 15.0 80 7.7 78 40.0 80 51.3 78 ns ns 
development missing 


Motivation to take a college course in the 50.8 65 39.1 64 54.2 48 30.8 39 ns * 
next year high or very high 


Motivation to take a college course in the 18.8 80 17.9 78 40.0 80 50.0 78 ns ns 
next year missing 


Motivation to take a college course in the 56.9 65 44.4 63 54.2 48 41.0 39 ns ns 
next five years high or very high 


Motivation to take a college course in the 18.8 80 19.2 78 40.0 80 50.0 78 ns ns 
next five years missing 


Motivation to earn a degree in the next five 61.5 65 56.3 64 56.3 48 43.6 39 ns ns 
years high or very high 


At time of assignment At end of study period 


Treatment (n = 80) Control (n = 78) Treatment (n = 80) Control (n = 78) Significance 


At time of At end of 
Outcome Percent Number Percent Number Percent Number Percent Number assignment study period 


Motivation to earn a degree in the next five 18.8 80 17.9 78 40.0 80 50.0 78 ns ns 
years missing 


Sense of identification as a professional: 57.5 73 67.1 73 72.9 48 56.4 39 ns ns 
strongly agree 


Identification as a professional missing 8.7 80 6.4 78 40.0 80 50.0 78 ns ns 
Increase in identification as a professional na na na na 18.2 44 16.2 37 na ns 
Increase in identification as a professional na na na na 45.0 80 52.6 78 na ns 
missing 


* Significant at p < .05. 

na is not applicable. ns is not significant. 

Note: No differences between groups for the confirmatory and secondary outcomes in the automatic scholarship enrollment randomization were significant at p < .01. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table B9. Demographic characteristics of Oregon’s early childhood education workforce and study samples, 
2018 (percent, unless otherwise indicated) 


State Sign-up randomization Incentive randomization 

DY-Tantoy-4e-] 0) alone a-la-led-la idle Number? Number? Mean Number? Mean 
Female 18,218 94.1 186 93.5 235 97.4 
Race/ethnicity 

White 17,604 69.6 253 64.0 230 64.3 

Latinx 17,604 19.0 253 22.5 230 25.7 

Neither White nor Latinx 17,604 10.6 253 13.4 230 10.0 
Age (years) 24,157 37.2 289 36.2 235 42.6 
Primary language 

English 18,963 85.2 193 84.5 235 81.7 

Non-English 18,963 14.8 193 15.5 235 18.3 
Highest level of education 

High school diploma or less 17,797 30.7 272 26.5 232 18.5 

Some college or other 17,797 23.1 272 23.9 232 46.6 

professional certificate 

Associate degree 17,797 13.5 272 11.0 232 12.9 

Bachelor’s degree or higher 17,797 32.7 272 38.6 232 22.0 


Job characteristics 


Workplace role 


Less than a lead teacher 24,203 70.2 242 78.5 219 41.6 

Lead teacher 24,203 11.3 242 13.6 219 22.4 

Supervisor or above 24,203 18.6 242 7.9 219 36.1 
Workplace type 

Certified child care center 24,203 77.0 242 87.6 216 67.1 

Not certified child care center 24,203 23.0 242 12.4 216 32.9 


Missing records 


Gender missing 24,203 24.7 289 35.6 235 0.0 
Race/ethnicity missing 24,203 27.3 289 12.5 235 2.1 
Age missing 24,203 0.2 289 0.0 235 0.0 
Primary language missing 24,203 21.7 289 33.2 235 0.0 
Self-reported education missing 24,203 26.5 289 5.9 235 1.3 
Workplace role missing 24,203 0.0 289 16.3 235 6.8 
Workplace type missing 24,203 0.0 289 16.3 235 8.1 


Note: Total possible records were 24,203 for state data, 289 for sign-up randomization, and 235 for incentive randomization. Group percentages might not 
sum to 100 percent because of rounding. 

a. Refers to the number of nonmissing values available as the denominator to calculate the percentage of the group with the listed characteristic. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018; state figures are from Oregon Center 
for Career Development in Childhood Care and Education and Oregon Childcare Research Partnership (2019). 
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Table B10. Demographic characteristics of early childhood education workforce members not in the Oregon 
early childhood education career lattice and individuals at step 7, 7.5, or 8 in the career lattice compared with 
study samples, 2018 

Panel A. Workforce members not in the Oregon early childhood education career lattice (sign-up randomization) 


Si-4anie) om eel ae (olanirz-yare)a} 


1BY=Yaatoyz4e-] eal (omelat-le-leik-la idle Number? Mean 


Race/ethnicity 
White 2,300 69.3 253 64.0 
Latinx 2,300 18.3 253 22.5 
Neither White nor Latinx 2,300 12.4 253 13.4 
Primary language 
English 2,499 89.4 193 84.5 
Non-English 2,499 10.6 193 15.5 
Highest level of education 
High school diploma or less 2,128 52.2 272 26.5 
Some college or other professional certificate 2,128 6.3 272 23.9 
Associate degree 2,128 10.7 272 11.0 
Bachelor’s degree or higher 2,128 30.8 272 38.6 


Job characteristics 


Workplace role 


Less than a lead teacher 5,617 91.9 242 78.5 

Lead teacher 5,617 2.2 242 13.6 

Supervisor or above 5,617 5.9 242 7.9 
Workplace type 

Certified child care center 5,617 80.4 242 87.6 

Not certified child care center 5,617 19.6 242 12.4 


Missing records 


Race/ethnicity missing 5,617 59.0 289 12.5 
Primary language missing 5,617 55.5 289 33.2 
Self-reported education missing 5,617 62.1 289 5.9 
Workplace role missing 5,617 0.0 289 16.3 
Workplace type missing 5,617 0.0 289 16.3 
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Panel B. Workforce members at step 7, 7.5, or 8 in the career lattice (incentive randomization) 


Incentive randomization sample 


Demographic characteristic Number? Number? Mean 


Race/ethnicity 
White 1,705 63.1 230 64.3 
Latinx 1,705 27.6 230 25.7 
Neither White nor Latinx 1,705 9.4 230 10.0 
Primary language 
English 1,715 77.3 235 81.7 
Non-English 1,715 22.7 235 18.3 
Highest level of education 
High school diploma or less 1,382 48.8 232 18.5 
Some college or other professional certificate 1,382 22.3 232 46.6 
Associate degree 1,382 10.0 232 12.9 
Bachelor’s degree or higher 1,382 19.0 232 22.0 
Job characteristics 
Workplace role 
Less than a lead teacher 1,756 65.0 219 41.6 
Lead teacher 1,756 10.8 219 22.4 
Supervisor or above 1,756 24.3 219 36.1 
Workplace type 
Certified child care center 1,756 77.0 216 67.1 
Not certified child care center 1,756 23.0 216 32.9 
Missing records 
Race/ethnicity missing 1,756 2.9 235 2.1 
Primary language missing 1,756 2.3 235 0.0 
Self-reported education missing 1,756 21.3 235 1.3 
Workplace role missing 1,756 0.0 235 6.8 
Workplace type missing 1,756 0.0 235 8.1 


Note: Data were not available for all eligible members of the population. Total possible records were 5,617 for state data in panel A, 289 for sign-up 
randomization in panel A, 1,756 for state data in panel B, and 235 for incentive randomization in panel B. Group percentages might not sum to 100 percent 
because of rounding. 

a. Refers to the number of nonmissing values available as the denominator to calculate the percentage of the group with the listed characteristic. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education (OCCD) data from 2018; state figures were reported 
by OCCD to the study team as of 2018. 


Table B11. Power calculations for study sample sizes 


W/Haliaalelaaime(siacveir-]e)( = 
effect size 


R- Sample 
(O]U} ereaat=) squared (om O KO )o) a=0.1 size 


Sign-up randomization 


Enrolled in career lattice between 10/12/2018 and 12/31/2019 0.1030 0.278 0.237 289 
Retained 0.0909 0.284 0.242 280 
Incentive randomization: Monetary incentive 

Increased at least one step in the career lattice since 9/1/2018 0.2716 0.343 0.292 155 
College credit hours earned between 9/1/2018 and 12/31/2019 0.1837 0.363 0.309 155 
Total community-based training hours recorded between 9/1/2018 and 0.2235 0.354 0.302 155 
12/31/2019 

Retained 0.2482 0.353 0.301 151 
Incentive randomization: Automatic scholarship enrollment 

Received a community college scholarship 0.2143 0.352 0.301 158 
Increased at least one step in the career lattice between 9/1/2018 and 0.2783 0.338 0.288 158 
12/31/2019 

College credit hours recorded between 9/1/2018 and 12/31/2019 0.2246 0.350 0.299 158 
Retained 0.2162 0.354 0.302 156 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Attrition. In the follow-up survey the attrition rate was 61 percent for the sign-up randomization and 40 percent 
for the combined arms of the incentive randomization (table B12). Differential attrition was 2 percent for the sign- 
up randomization and 13 percent for the incentive randomization. The high overall attrition rates for both 
randomizations and the differential attrition rate for the incentive randomization mean that less reliance can be 
placed on results using data from the follow-up survey. Under What Works Clearinghouse guidelines on attrition 
and optimistic assumptions that the treatment is unrelated to attrition for either randomization, the threat of bias 
is considered unacceptable, and attrition is considered high (U.S. Department of Education, 2017). 


Table B12. Overall attrition on the follow-up survey was high for both randomizations, and differential 
attrition was present for the incentive randomization 


Sign-up randomization Tatet=1aiah’{om a-darelo)aalr4-1a(e)a\sy 
Combined 
(Cfolatolr-Tasialle) 
Automatic FTare| 
scholarship Monetary monetary 
Control Treatment Total Control enrollment incentive incentive) a Kojw-}| 
Number of participants 149 140 289 78 80 77 157 235 
randomized 
Number of follow-up 56 56 112 40 49 52 101 141 
survey responses 
Response rate (percent) 38 40 39 51 61 68 64 60 
Overall attrition (percent) 62 60 61 49 39 32 36 40 
Differential attrition na na 2 na na na na 13 
(percent) 


na is not applicable. 

Note: Number of participants randomized is the number of participants allocated to the treatment and control groups, excluding those who were determined 
to be ineligible based only on characteristics determined prior to the intervention and applied in the same way to the treatment and control groups (such 
as already being in the career lattice for the sign-up randomization and being at a higher step for the incentive randomization). 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 
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Methodology 


The study team examined the impact of the interventions by comparing outcomes for ECE professionals between 
the treatment and control groups for each trial separately. Among significant findings, meaningful differences in 
outcomes were those greater than 0.20 standard deviation (following broad benchmarks for intervention effect 
sizes, as specific benchmarks for early educator interventions are not established in the literature; Cohen, 1988). 
Estimates are reported as statistically significant if below the .10 level to account for lower power to detect effects 
due to smaller sample sizes. 


The main and secondary outcomes were analyzed for both randomizations. Main outcome data had little or no 
missing information, while secondary outcomes were from survey data with missing information, which caused 
challenges for the analysis (see box 2 in the main report and the “Data” section earlier in this appendix for more 
information on missing data). As described in the “Key variables and outcome measures” section above, the main 
outcomes for the sign-up intervention were an indicator for career lattice sign-up and an indicator of retention at 
the same workplace during the study period. The main outcomes for the incentive randomization differed for the 
two treatments. For the case of the automatic scholarship enrollment, the main outcomes were indicators for 
using a scholarship, increasing at least one step in the career lattice, college credit hours earned, and retention 
since randomization. The main outcomes for the monetary incentive treatment were the number of professional 
development hours recorded, the number of college credits earned, an increase of at least one step in the career 
lattice, and workplace retention since randomization. Secondary outcomes are described in the “Exploratory 
analysis” section below. 


The sign-up randomization was conducted at the individual level, and all analyses were also conducted at the 
individual level. The incentive randomization was conducted at the workplace level, but the analysis was 
conducted at the individual level due to very little clustering of individuals within the workpalce.? The impact of 
receiving the nudge, known as the intent-to-treat (ITT) effect in econometric terminology, was estimated for each 
treatment group and randomization. For binary outcomes, such as indicators for enrollment in the career lattice, 
a linear probability model was used. 


For research question 1 the ITT effect of the emails (to sign up for the career lattice) on the binary main outcome 
(the indicator for career lattice sign-up) was estimated with the following model: 


Y¥; = Po + B,Treat; + XO +e; (B1) 


where Y is the binary outcome measure for educator i, Treat is an indicator variable taking a value of 1 for the 
treatment group and 0 for the control group, ¢ is the error term that is assumed to be distributed N(0,07), and X 
represents a vector of control variables as summarized in table B2. The ITT effect is given by 6, and is the 
difference in the outcome measure Y between educators who were randomly assigned to receive the nudge and 
those who were randomly assigned to not receive the nudge. 


Due to the small sample sizes and interest in ensuring balance between the treatment and control groups for 
many covariates, a rerandomization procedure was employed. This means that the study team set a threshold of 
acceptance for the difference in specific treatment and control covariates,’ ran repeated randomizations, and 


? Standard errors were not clustered at the workplace level due to the few instances of clustering in the data. 

3 The covariates for the sign-up randomization were education level, race/ethnicity, years planned to continue in the workforce, age of 
children served, workplace type, whether planned to continue professional development, whether planned to continue education, strongly 
agree to identification as a professional, has the Child Development Associate credential, and missing response to being motivated to 
continue education and training. For the incentive randomization, the covariates were: step level, path in the career lattice, education level, 
age of children served, workplace type, standardized measure of years working, years planned to continue in the workforce, whether an 
increase in education or training would lead to a raise, when they last enrolled in a college course, planned to continue education, workplace 
very supportive of increasing education and training, and race/ethnicity. 
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randomly assigned the treatment condition to the units only after the threshold was met. This method not only 
permitted analysis of the findings without controls for baseline characteristics but also confirmed that the baseline 
characteristics used in the rerandomization procedure were indeed balanced in the analytic sample. Proper 
analysis of rerandomized trials involved calculating an adjusted p-value because analyzing rerandomized trials 
using standard methods resulted in a p-value that is too conservative. The adjusted p-value was estimated using 
randomization inference tests (Mihaly et al., forthcoming; Morgan & Rubin, 2015). Randomization inference 
considers what would have occurred under all possible random assignments and is performed at the analysis 
stage. The steps to randomization inference involve a Monte Carlo framework: 


1. Preserve the original treatment assignment. 
2. Generate placebo treatment statuses according to rerandomization assignment method. 


3. Estimate the original regression equation substituting the original treatment assignment variable with a 
term for the placebo treatment. 


4. Repeat steps 1-3 1,000 times. 


5. The randomization inference p-value is the proportion of times the placebo treatment effect was larger 
than the estimated treatment effect. 


The impact of the monetary incentive or automatic scholarship enrollment emails on binary outcomes (incentive 
randomization) for research questions 2 and 3 were modeled in the same way as described in equation B1. For 
continuous outcomes for research questions 1, 2, and 3, the outcome measure in equation B1 was replaced with 
Y, the continuous outcome, and the model was estimated using ordinary least squares. 


Sensitivity analysis. The study team conducted sensitivity analyses to test the extent to which estimates were 
driven by model assumptions and to examine whether estimating the models including educator covariates would 
affect the findings. A missing indicator method was used to account for covariates that had missing data. The 
results were not sensitive to the set of control variables used. 


Exploratory analysis. The exploratory analysis of secondary outcomes was conducted according to the models 
described above and included secondary outcomes from the surveys, where response rates were low for both 
randomizations, and treatment and control groups had different response rates for the incentive randomization. 
Outcomes drawn from the survey differed for each trial. 


For the sign-up randomization the survey-derived outcome was: 

e Reported increase in identification as a professional. 

For the monetary incentive (incentive randomization) the survey-derived outcomes were: 

e Reported motivation to take college coursework or complete a certification or degree. 

e Reported increase in identification as a professional. 

For the automatic scholarship enrollment (incentive randomization) the survey-derived outcomes were: 
e Reported motivation to take college coursework or complete a certification or degree. 

e Reported increase in identification as a professional. 


The study team conducted exploratory analyses on groups of participants to gain a better understanding of the 
heterogeneity of the effects of nudges and incentive changes. They examined the differential impact of the 
interventions by individual factors, including age of the study participant (broken into three categories: younger 
than 35, 35-54, and 55 and older) at baseline; primary language at baseline (English or non-English); workplace 
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role at baseline (teacher, director, or owner compared with assistant teachers and aides); education level at 
baseline (associate degree or higher compared with lower than an associate degree); workplace at baseline 
(home-based care compared with center-based care, Head Start, preschool programs, and other types); 
race/ethnicity (educators of color compared with White educators); urban or rural location; and distance to the 
nearest Oregon public college or university (less than 5 miles, 5—20 miles, and more than 20 miles). 


To estimate the group effects in the case where the group had two categories, equation B1 was modified to 
include an interaction term for treatment status and the group of interest s = 1,...,S: 


Y = Bo t+ ByTreat * (X,, = 1) + B2Treat * (X,, = 0) + X0+ ¢; (B2) 


where (X,; = 1) indicates that the group is true and (X,; = 0) indicates that the group is false. For cases in which 
the group contained more than two categories, the model included an interaction term for the group in each of 
the possible scenarios with the treatment effect. All group analyses were conducted separately for each group. 


For research question 4 the study team calculated descriptive statistics of survey questions and email platform 
data by intervention and treatment group status to report counts and percentages for items of interest. For the 
OCCD interview data formal theme coding was not conducted because of the small number of interviews; rather, 
the study team reported where an informant had specific information about an implementation topic and where 
consensus emerged across informants. 


Missing data. Prior to starting the analyses, the study team examined the extent of missing data overall and by 
treatment group. Because the main outcomes were from administrative data, there were no missing data in those 
analyses. The secondary outcomes were from follow-up surveys and did suffer from high overall attrition for both 
randomizations and from differential attrition for the incentive randomization. However, because these were 
exploratory analyses and because of the low overall response rates to the follow-up survey, no procedures were 
implemented to account for missing outcome data for secondary outcomes. 


Crossovers. The ITT analysis described above does not take into account whether educators received the emails 
for any of the three interventions across the two trials. It is possible that some treatment group educators did not 
receive the emails (for example, the email ended up in a junk email folder) or that some control group educators 
found out about the emails from colleagues. Because the study team was interested not only in whether the policy 
of sending emails was effective (as measured by the ITT effect) but also in whether the email itself was effective 
for the people who receive it, exploratory models of the treatment-on-the-treated (TOT) impact of the emails 
were estimated. An instrumental variable analysis was used to estimate the TOT effect. Specifically, the following 
pair of equations were estimated using the two-staged least squares methodology: 


D; = 6Treat; + Li (B3) 
Y =6D;+ XO+ (B4) 


where D is an indicator for whether educator j received the email, Y is the binary outcome measure, and X and € 
are as described above. Several options were considered to measure receipt of the email because there was a rich 
set of data on whether the respondent opened the email that was sent and whether the respondent clicked on 
any links within the email. For the analysis presented in the report, the first stage outcome measure D is an 
indicator for whether respondents opened and clicked on a link in at least one of the emails they received about 
the intervention. The random assignment to the treatment group (Treat) was used as an instrumental variable for 
having received the email in the second stage. The coefficient of interest was 6, which is the regression-adjusted 
estimate of the TOT effect. 
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Protocols 


Sign-up randomization: Baseline survey protocol 


What is your preferred email address?* 


What is your secondary email address? 


1. What is your full name (first and last name)? 


2. What is your birthdate? MM/DD/YYYY? ___ (Month) __ (Day) (Year) 


3. What is the ZIP code of your primary residence? 


4. What is the highest level of education you have achieved? 
e No high school diploma or GED 

e High school diploma or GED 

e Some college but no degree or certificate 

e Associate degree 

e Bachelor’s degree 

e Master’s degree 

e Higher than a master’s degree 


5. Do you have a Child Development Associate (CDA) certification? 
e =6Yes 

e No 

e =Don’t know 


[skip logic: Only show if no to CDA question above] 


6. Are you planning to get a Child Development Associate (CDA) certification? 
e §6Yes 

e No 

e Don’t know 


7. Do you plan to continue your formal education (such as taking college classes)? 
e §=6Yes 

e No 

e = Don’t know 


[skip logic: Only show if answer yes to formal education question above] 


8. What is your educational goal? 

e Additional college classes but no degree 
e Associate degree 

e Bachelor’s degree 

e Other (please specify) 
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Do you plan to participate in professional development for which you will not receive college or university 
course credit? 

Yes 

No 

Don’t know 


[skip logic: show if yes to continued education and/or training hours] 


10. 


What motivates you to continue your education and training? Please check all that apply. 
Meet Office of Child Care licensing requirements 

My workplace requires me to continue education and training 

Get a job 

Receive a promotion/pay raise 

Scholarship money 

Personal growth 

Spark requirements 

Other (please specify) 


[skip logic: show if no to both questions on continuing education and training hours] 


11. 


What would motivate you to continue your education and training? Please check all that apply. 
Meet Office of Child Care licensing requirements 

My workplace requires me to continue education and training 

Get a job 

Receive a promotion/pay raise 

Scholarship money 

Personal growth 

Spark requirements 

Other (please specify) 


. What kind(s) of professional development do you find most helpful for learning new information/skills 


and applying knowledge in your work? Please select your top three types of professional development 
from the list below. 

Being observed and receiving feedback from another early learning professional 
Visiting or observing other child care classrooms 

Participating in online or distance learning (web-based classes, etc.) 

Taking college classes in a cohort group (e.g., a group that starts classes together) 
Taking college classes on your own 

Attending conferences 

Participating in a learning community with other early learning professionals 
Other (please specify) 

Other (please specify) 

Other (please specify) 


The next set of questions will ask you about your workplace. If you work in more than one setting, please answer 
the following questions for the place where you work the most hours per week. If you work an equal number of 
hours at two or more workplaces, please pick one workplace for your responses. 
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13. 


Which best describes the program you work in? 
Head Start (excluding Early Head Start) 

Early Head Start 

Child care center 

Home-based care 

Relief Nursery 

Other (please specify): 


. What ages of children do you have in your program? Please check all that apply. 


Infant 

Toddler 

Preschool 

School-age (ages 5 to 12) 
Don’t know 


. What is your current position? Please check the option that most closely matches your main role at your 


workplace. 

Director 
Provider/owner 

Lead /head teacher 
Teacher 

Assistant teacher 
Aide/assistant 

Other (please specify): 


[skip logic: only show if answered director or provider/owner in prior question on position) 


16. 


17. 


Does your workplace receive Preschool Promise funding? 
Yes 

No 

Don’t know 


How many years have you been working in this field (with children aged 0-5)? Only count work in the 
childhood care and education field, which includes the following settings: certified family child care; 
registered family child care; certified child care center; for-profit, not-for-profit, or faith-based program; 
early intervention/early childhood special education; teen parent or relief nursery program; child care 
resource and referral agency; Head Start; state agency; consulting business; professional organization; 
higher education 


. How long do you plan to continue working in this field (with children aged 0-5)? 


Less than 1 year 

1-2 years 

3-5 years 

More than five years 
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[skip logic: Only show if chose anything but More than 5 years in question above about years continue working] 


19. Why do you plan to leave this field? [write in] 


20. How strongly do you agree or disagree with the phrase “I feel like | am an early learning professional” ? 


e Strongly agree 

e Somewhat agree 

e Neither agree nor disagree 
e Somewhat disagree 

e Strongly disagree 


21. What is your current hourly wage at your primary early learning job? 
e Less than $10.50 

e $10.50 to $11.99 

e $12.00 to $12.99 

e $13.00 to $15.99 

e $16.00 to $19.99 

e $20.00 or higher 

e Don’t know 


22. For this study, OCCD will be contacting you through email. How often do you usually check your primary 
email? Please select the closest match. 


e Hourly 
eDaily 

e Weekly 
e Monthly 


Finally, we want to ask you about your background to help us understand more about you. 


23. Which race/ethnicity do you identify with? Please check all that apply. 
e Don’t want to answer 

e American Indian or Alaska Native 
e Hispanic or Latino/a 

e = Asian 

e Native Hawaiian or Pacific Islander 
e Black or African American 

e Middle Eastern/Northern African 
e White 

e Other (please specify) 

e Don’t know 


24. Any last comments or thoughts for us? 
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Sign-up 


[Follow- 


REL 2021 


randomization: Follow-up survey protocol 
What is your full name (first and last name)? 


What is the highest level of education you have achieved? 
No high school diploma or GED 

High school diploma or GED 

Some college but no degree or certificate 

Associate degree 

Bachelor’s degree 

Master’s degree 

Higher than a master’s degree 


Do you plan to continue your formal education (such as taking college classes)? 
Yes 

No 

Don’t know 


Do you plan to participate in professional development for which you will not receive college or university 
course credit? 

Yes 

No 

Don’t know 


How strongly do you agree or disagree with the phrase “I feel like | am an early learning professional”? 


Strongly agree 

Somewhat agree 

Neither agree nor disagree 
Somewhat disagree 
Strongly disagree 


up only questions (new questions)] 


Do you know about the Oregon Registry Steps? 
Yes 

No 

Not sure 


Do you know about the education awards offered by OCCD for advancing to certain Steps in the Oregon 
Registry? 

Yes 

No 

Not sure 


Did you receive any emails encouraging you to sign up for the Oregon Registry Steps since August 2018? 
Yes 

No 

Don’t know 
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[skip logic: show if Yes or Don’t know on Q7] 


9. Did you sign up for the Oregon Registry Steps? 
e §=6Yes 

e No 

e Don’t know 


[skip logic: Yes on Q8 and Q9 show Q10] 


10. Did receiving an email encouraging you to sign up for the Oregon Registry Steps motivate you to sign up? 
e §6Yes 

e No 

e Don’t know 


[skip logic: Yes on Q9 show Q11] 


11. Why did you sign up for the Oregon Registry Steps? Please select all that apply. 
e Toreceive an education award 

e Tobe eligible for scholarships 

e Tobe recognized for my educational achievements 

e Tohelp me plan my professional development 

e Toconnect to others who care about children and families 

e Myemployer required me 

e Other (please specify): 


[skip logic: Yes on Q9 show Q12] 


12. How much has a Step on the Oregon Registry helped you ... 
a. Feel more professional? 

b. Plan your professional development? 

c. Connect to others who care about children and families? 

e Helped a lot 

e Helped a little 

e Did not help at all 


[skip logic: No on Q9 show Q13] 


13. Why did you choose not to sign up for the Oregon Registry Steps? Please select all that apply. 
e Didn’t want to spend time signing up 

e Not planning to continue in the field 

e Not sure how to sign up 

e Don’t have the necessary paperwork 

e Don’t know about the Oregon Registry Steps 

e Other (please specify): 


[End follow-up section of new questions] 


The next set of questions will ask you about your workplace. If you work in more than one setting, please answer 
the following questions for the place where you work the most hours per week. If you work an equal number of 
hours at two or more workplaces, please pick one workplace for your responses. 
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14. What is your current position? Please check the option that most closely matches your main role at your 
workplace. 

e Director 

e ~=Provider/owner 

e Lead /head teacher 

e Teacher 

e =6Assistant teacher 

e = Aide/assistant 

e Other (please specify): 


15. How long do you plan to continue working in this field (with children aged 0-5)? 
e Less than 1 year 

e 1-2 years 

e 3-5 years 

e More than 5 years 


16. What is your current hourly wage at your primary early learning job? 
e —_ Less than $10.50 

e $10.50 to $11.99 

e $12.00 to $12.99 

e $13.00 to $15.99 

e $16.00 to $19.99 

e $20.00 or higher 

e Don’t know 


17. Any last comments or thoughts for us? 


Incentive randomization: Baseline survey protocol 
1. What is your full name (first and last name)? 


2. What is your birthdate? MM/DD/YYYY? ___ (Month) __ (Day) (Year) 


3. What is the ZIP code of your primary residence? 


4. What is the highest level of education you have achieved? 
e No high school diploma or GED 

e High school diploma or GED 

e Some college but no degree or certificate 

e Associate degree 

e Bachelor’s degree 

e Master’s degree 

e Higher than a master’s degree 


5. Do you have a Child Development Associate (CDA) certification? 
e §=6Yes 

e No 

e Don’t know 
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[Skip logic: Only show if no to CDA question above] 


Are you planning to get a Child Development Associate (CDA) certification? 
Yes 

No 

Don’t know 


The next set of questions will ask you about your workplace. If you work in more than one setting, please answer 
the following questions for the place where you work the most hours per week. If you work an equal number of 
hours at two or more workplaces, please pick one workplace for your responses. 


Which best describes the program you work in? 
Head Start (excluding Early Head Start) 

Early Head Start 

Child care center 

Home-based care 

Relief nursery 

Other (please specify) 


What ages of children do you have in your program? Please check all that apply. 
Infant 

Toddler 

Preschool 

School-age (ages 5 to 12) 

Don’t know 


What is your current position? Please check the option that most closely matches your main role at your 
workplace. 

Director 

Provider/owner 

Lead /head teacher 

Teacher 

Assistant teacher 

Aide/assistant 

Other (please specify): 


[Skip logic: only show if answered director or provider/owner in prior question on position) 


10. 


11. 


Does your workplace receive Preschool Promise funding? 
Yes 

No 

Don’t know 


How many years have you been working in this field? (with children aged 0-5)? Only count work in the 
childhood care and education field, which includes the following settings: certified family child care; 
registered family child care; certified child care center; for-profit, not-for-profit, or faith-based program; 
early intervention/early childhood special education; teen parent or relief nursery program; child care 
resource and referral agency; Head Start; state agency; consulting business; professional organization; 
higher education 
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. How long do you plan to continue working in this field (with children aged 0-5)? 


Less than 1 year 
1-2 years 

3-5 years 

More than 5 years 


[skip logic: only show if chose anything but more than five years in question above about years continue working] 


13. 


14. 


Why do you plan to leave this field? [write-in] 


How strongly do you agree or disagree with the phrase “I feel like | am an early learning professional”? 


Strongly agree 

Somewhat agree 

Neither agree nor disagree 
Somewhat disagree 
Strongly disagree 


. What is your current hourly wage at your primary early learning job? 


Less than $10.50 
$10.50 to $11.99 
$12.00 to $12.99 
$13.00 to $15.99 
$16.00 to $19.99 
$20.00 or higher 
Don’t know 


. Would you expect to receive a raise at your current job if you earned a higher educational degree than 


you currently have? 
Yes 

No 

Don’t know 


. When did you last take a college course? A college course could be at a community college or university. 


lam currently enrolled in a college course for Fall 2018 

Within the last year (2017 — 2018 school year) but not currently 
Within the last three years 

Within the last five years 

Within the last ten years 

More than ten years ago 


. When did you last participate in professional development for which you did not receive college or 


university course credit? 

lam currently taking professional development 
Within the last 12 months but not currently 
1-2 years ago 

More than 2 years ago 
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19. What is your level of motivation to take a college course in the next 12 months? 
e Very low 


e «Low 

e Moderate 
e =High 

e Very high 


20. What is your level of motivation to take a college course in the next 5 years? 
e Very low 


e Low 
e Moderate 
e =High 
e Very high 


21. What is your level of motivation to earn a higher educational degree or early childhood education-related 
certification in the next 5 years? 
e Very low 


e Low 
e Moderate 
e =High 
e Very high 


22. Do you plan to continue your formal education (such as taking college classes)? 
e §=6Yes 

e No 

e Don’t know 


[skip logic: only show if answer yes to formal education question above] 


23. What is your educational goal? 

e Additional college classes but no degree 
e Associate degree 

e Bachelor’s degree 

e Other (please specify) 


24. Do you plan to participate in professional development for which you will not receive college or university 
course credit? 

e §=6Yes 

e No 

e Don’t know 
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[skip logic: show if yes to continued education and/or training hours] 


25. 


What motivates you to advance your education? Please check all that apply. 
Meet Office of Child Care licensing requirements 

My workplace requires me to continue education and training 

Get a job 

Receive a promotion/pay raise 

Scholarship money 

Personal growth 

Spark requirements 

Other (please specify) 


[skip logic: show if no to both questions on continuing education and training hours] 


26. 


What would motivate you to advance your education? Please check all that apply. 
Meet Office of Child Care licensing requirements 

My workplace requires me to continue education and training 

Get a job 

Receive a promotion/pay raise 

Scholarship money 

Personal growth 

Spark requirements 

Other (please specify) 


. What kind(s) of professional development do you find most helpful for learning new information/skills 


and applying knowledge in your work? Please select your top three types of professional development 
from the list below. 

Being observed and receiving feedback from another early learning professional 
Visiting or observing other child care classrooms 

Participating in online or distance learning (web-based classes, etc.) 

Taking college classes in a cohort group (e.g., a group that starts classes together) 
Taking college classes on your own 

Attending conferences 

Participating in a learning community with other early learning professionals 
Other (please specify) 

Other (please specify) 

Other (please specify) 


. What factors affect your decision to continue your education? Please rank your selections with the most 


important factor ranked first. You do not need to rank all the options provided. 
Cost 

Distance 

Time 

Difficulty of subject matter 

Family commitments 

Language barrier 

Other 
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29. 


30. 


31. 


If you selected ‘Other’ in the previous question, please describe the other factors that affect your decision 
to continue your education. [open-ended] 


What support would best help you connect to formal education and professional development? [open- 
ended] 


How supported do you feel by your workplace to participate in ongoing professional development? 
Very supported 

Somewhat supported 

Not supported 


. Are you or were you part of a Child Care Network (a cohort learning system for home-based child care 


providers)? 
Yes 
No 
Don’t know 


. For this study, OCCD will be contacting you through email. How often do you usually check your primary 


email? Please select the closest match. 
Hourly 

Daily 

Weekly 

Monthly 


Finally, we want to ask you about your background to help us understand more about you. 


35. 


. With which race/ethnicity do you identify with? Please check all that apply. 


Don’t want to answer 

American Indian or Alaska Native 
Hispanic or Latino/a 

Asian 

Native Hawaiian or Pacific Islander 
Black or African American 

Middle Eastern/Northern African 
White 

Other (please specify) 

Don’t know 


Any last comments or thoughts for us? 
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Incentive randomization: Follow-up survey protocol 


What is your full name (first and last name)? 


What is the highest level of education you have achieved? 
No high school diploma or GED 

High school diploma or GED 

Some college but no degree or certificate 

Associate degree 

Bachelor’s degree 

Master’s degree 

Higher than a master’s degree 


How strongly do you agree or disagree with the phrase “I feel like | am an early learning professional”? 


Strongly agree 

Somewhat agree 

Neither agree nor disagree 
Somewhat disagree 
Strongly disagree 


What is your level of motivation to take a college course in the next 12 months? 
Very low 

Low 

Moderate 

High 

Very high 


What is your level of motivation to take a college course in the next 5 years? 
Very low 

Low 

Moderate 

High 

Very high 


What is your level of motivation to earn a higher educational degree or early childhood education-related 
certification in the next 5 years? 

Very low 

Low 

Moderate 

High 

Very high 


Follow-up only 


Did you take any college or university courses since August 2018? 
Yes 

No 

Don’t know 
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[skip logic: show Q8 if yes on Q7] 


Were any of these courses on a semester schedule? 

Yes, some or all of my courses were on a semester schedule 
No, all of my courses were on a quarter schedule 

Don’t know 


[skip logic: show Q9-Q10 if yes on Q7] 


9. 


How many fall 2018 college or university courses did you complete? 


[skip logic: show Q10 if Q9 is greater than 0] 


10. 


How many credits did you earn? 


[skip logic: show Q11 if Q8 if yes or skipped and Q9 is greater than 0] 


11. 


12. 


Were these courses on a quarter schedule or semester schedule? 
Quarter 
Semester 


Do you plan to take any fall 2019 college or university courses? 
Yes 

No 

Don’t know 


. When did you last participate in professional development for which you did not receive college or 


university course credit? 

lam currently taking professional development 
Within the last 12 months but not currently 
1-2 years ago 

More than 2 years ago 


[skip logic: show Q14 if Q13 is currently or within last 12 months] 


14. 


15. 


Thinking about all of the professional development that you participated in from August 2018 through 
August 2019, approximately how many total hours did you participate? 


Do you plan to participate in professional development for which you will not receive college or university 
course credit? 

Yes 

No 

Don’t know 


. Do you know about the education awards offered by OCCD for advancing to certain Steps in the Oregon 


Registry? 
Yes 
No 
Not sure 
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[skip logic: show if yes to Q16] 


17. 


To what extent do the education awards offered through OCCD motivate you to enroll in college courses, 
take professional development courses, or earn a degree or certificate? 

To a large extent 

To a moderate extent 

To a small extent 

Not at all 


. Did you receive any emails about eligibility for a special education award program for moving to your next 


Step on the Oregon Registry Steps since August 2018? 
Yes 

No 

Don’t know 


[skip logic: Only show Q19 if answered yes in Q18] 


19. 


Did the email about eligibility for a special education award program for moving to your next Step on the 
Oregon Registry Steps motivate you to complete professional development or college coursework? 

Yes 

No 

Don’t know 


[skip logic: show if yes to Q16] 


20. 


Since August 2018, did you apply for an education award through OCCD for advancing a Step in the Oregon 
Registry? 

Yes 

No 

Don’t know 


[skip logic: Show if no on Q20] 


21. 


coo 


Why did you choose not to apply for an education award from OCCD? Please check all that apply. 
Unsure of application process 

Did not qualify for an award since August 2018 

Not interested in taking more courses or training needed to move up a Step 

Didn’t want to deal with paperwork 

Didn’t want to provide information specific to the W-9 form in the application 

Award amount wasn’t enough to interest me 

Other (please specify): 


. How much has a Step on the Oregon Registry helped you... 


Feel more professional? 

Plan your professional development? 

Connect to others who care about children and families? 
Helped a lot 

Helped a little 

Did not help at all 
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23. 


Do you know about the scholarships offered by OCCD to help you pursue additional training and 
education? 

Yes 

No 

Unsure 


[skip logic: show if yes to Q23] 


24. 


To what extent do the scholarships offered through OCCD motivate you to enroll in college courses, take 
professional development courses, or earn a degree or certificate? 

To a large extent 

To a moderate extent 

To a small extent 

Not at all 


. Did you receive an email about being automatically enrolled in a scholarship program from OCCD since 


August 2018? 
Yes 

No 

Don’t know 


[skip logic: only show Q26 if answered yes in Q25] 


26. 


Did the email about being automatically enrolled in a scholarship from OCCD motivate you to access the 
scholarship funds? 

Yes 

No 

Don’t know 


[skip logic: show Q27 if yes on Q23] 


27. 


Did you try to access a scholarship from OCCD during the past year? 
Yes 

No 

Don’t know 


[skip logic: Only show Q28 if no to Q27] 


28. 


Why did you not try to access a scholarship from OCCD? Please check all that apply. 
Not interested in taking college courses 

Don’t have the time to take college courses 

Don’t have financial need 

Didn’t want to deal with paperwork 

Unable to get my supervisor to sign the application 

The college wouldn’t get the scholarship money in time to cover costs 

Scholarship amount wouldn’t cover enough of the costs of taking courses 

My college or university isn’t eligible for the scholarship 

Other (please specify): 


[End of follow up (new questions)] 
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The next set of questions will ask you about your workplace. If you work in more than one setting, please answer 
the following questions for the place where you work the most hours per week. If you work an equal number of 
hours at two or more workplaces, please pick one workplace for your responses. 


29. What is your current position? Please check the option that most closely matches your main role at your 
workplace. 

e = Director 

e =Provider/owner 

e Lead /head teacher 

e Teacher 

e =6Assistant teacher 

e = Aide/assistant 

e Other (please specify): 


30. How long do you plan to continue working in this field (with children aged 0-5)? 
e Less than 1 year 

e 1-2 years 

e 3-5 years 

e More than 5 years 


31. What is your current hourly wage at your primary early learning job? 
e Less than $10.50 

e $10.50 to $11.99 

e $12.00 to $12.99 

e $13.00 to $15.99 

e $16.00 to $19.99 

e $20.00 or higher 

e Don’t know 


32. Any last comments or thoughts for us? 


Oregon Center for Career Development in Childhood Care and Education staff interview protocol 


1. Please describe your involvement in the process OCCD used to send out the study information on 
education awards and scholarship enrollment. 


a. In addition to the information you have provided in the [documentation], is there anything else 
you’d like to add regarding the process used to send out the emails? What went well and what 
didn’t go as you had hoped? 


b. Probe: Which stakeholders were informed of the study? How were they informed? What was 
their involvement in the study or process of implementing the study? Do you think that other 
people would have heard about the study from these stakeholders? 


c. Probe: Were there any communications sent to the ECE workforce other than the survey 
invitations or email “nudge” messages (e.g., other stakeholders sent out messages or flyers)? 
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2. Were there any challenges or issues with this process (e.g., processing applications, bad email addresses; 
differences in ability to reach different workers by auspice, such as home-based providers; timeline 
constraints to stay on email reminders schedule)? 


a. Ifso, how were those issues addressed/resolved? 
3. Why do you think that some people chose to opt out [of the emails]? 
4. From your perspective, what do you believe was successful about: 

a. The email campaign? 

b. Administering the modified awards/scholarship program? 

c. OCCD’s participation in the study? 


5. Did you receive any comments or feedback from the ECE workforce in the study on the process (in addition 
to emails received)? 


a. Probe: Please provide specific examples of positive and negative feedback. (E.g., people liked the 
email style; people disliked getting reminder emails) 


6. Did you receive any comments or feedback from other Oregon stakeholders on the process? 


a. Probe: Please provide specific examples of positive and negative feedback (e.g., heard comment 
at a meeting). 


b. Probe: Please describe your outreach to policymakers/stakeholders about this intervention. What 
were their reactions? 


7. If you were to do this type of intervention again, is there anything you would do differently? 


a. In your opinion, given that we had to end earlier than planned, what is the ideal length of time 
that the email campaign could have lasted? 


b. How could the burden on OCCD staff have been lessened? 
8. How do you think the study could influence OCCD’s future work? What about state policy? 
a. Probe: Do you think you would do email campaigns again? For what purpose? 


9. Is there any other feedback you would like to share with us about the intervention/process? 
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Appendix C. Supporting analysis 


This appendix provides additional analysis results related to estimated impacts and survey responses to support 
the information in the body of the report. 


Sign-up randomization, additional analyses 


The tables below show the treatment-on-the-treated estimates (table C1), group analyses for the sign-up 
randomization (table C2), and the secondary outcomes (table C3). 


Table C1. Treatment-on-the-treated estimates of the sign-up email on career lattice sign-up and workplace 
retention during the study period, 2018 and 2019 


Treatment ero} ahiae)| Estimated 
group mean group mean Taal of-\ei a 
Gielaler-lae! Giclalek- rae! Gilatel-lae! Effect 
Outcome deviation) deviation) error’) size® 
Signed up for the career lattice 0.186 0.154 0.102 0.282 289 
between 10/12/2018 and 12/31/2019 (0.390) (0.363) (0.158) 
Workplace retention 0.803 0.811 —0.046 —0.116 280 
(0.399) (0.393) (0.168) 


Note: None of the effects was statistically significant at p < .10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were 
estimated. 

c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


The email encouraging career lattice sign-up had no detectable impact on specific populations. Sending emails 
had no detectable impact on career lattice sign-up or workplace retention by age range, primary language, 
education level, race/ethnicity, workplace role, workplace type, urban or rural location, or distance to the closest 
in-state public college or university (table C2). 
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Table C2. Group analyses of the sign-up email on enrollment in the career lattice and workplace retention during the study period, 2018 and 2019 


Group 2 Group 3 
Treatment (olaiage)| 
group group Estimated Estimated Estimated 
mean mean impact? impact? impact? 
(Gielarekyac} (Gielarerlae! (Gielarek-lac} (Gielarekyae} (Gielareklac} Sample 
Outcome deviation) deviation) error’) Effect size° error’) Effect size° error’) Effect size° size? 
Age group: Younger than 35 Age group: 35-54 Age group: 55 and older 

Signed up for the career lattice 0.186 0.154 0.081 0.223 -0.048 -0.132 0.008 0.022 289 
ees 12/2018 and (0.390) (0.363) (0.059) (0.059) (0.088) 
Workplace retention 0.803 0.811 —0.030 —0.076 0.050 0.127 —0.097 —0.248 280 

(0.399) (0.393) (0.060) (0.067) (0.110) 

Primary language: English Primary language: Non-English 

Signed up for the career lattice 0.283 0.228 0.010 0.023 0.139 0.331 na na 193 
See eS (0.453) (0.421) (0.070) (0.148) 
Workplace retention 0.730 0.758 -0.052 —0.122 0.058 0.135 na na 184 

(0.446) (0.431) (0.070) (0.141) 

Workplace role: Teacher or above Workplace role: Below teacher 

Signed up for the career lattice 0.195 0.169 0.036 0.096 0.011 0.030 na na 242 
See 2 20a (0.398) (0.377) (0.071) (0.058) 
12/31/2019 
Workplace retention 0.783 0.814 0.006 0.016 —0.058 -0.149 na na 233 

(0.414) (0.391) (0.069) (0.067) 

Education level: Education level: 
Associate degree or higher Below associate degree 

Signed up for the career lattice 0.194 0.167 0.038 0.100 0.019 0.050 na na 272 
ee Eye | (0.397) (0.374) (0.060) (0.060) 
Workplace retention 0.794 0.803 0.072 0.180 —0.107 —0.268 na na 263 


(0.406) (0.399) (0.054) (0.071) 


Group 1 Group 2 Group 3 


Treatment Control 
Agelvyo) figeleyo) Estimated Estimated Estimated 
mean mean Tan) ex-\el ta Ta) ex-\el ta impact? 
(Giclarerlae) (Giclaterlae) (Gielareklac} (Gieclaleklac} (Gieclaleklac} Sample 
Outcome deviation) deviation) Taney aa) Effect size® Taney an) Effect size® Taneyan) Effect size® sized 
Race/ethnicity: White Race/ethnicity: Other than White 

Signed up for the career lattice 0.202 0.178 0.031 0.081 0.011 0.030 na na 253 
peaion Mane (0.403) (0.384) (0.057) (0.072) 
12/31/2019 
Workplace retention 0.787 0.789 —0.005 —0.012 0.003 0.006 na na 245 

(0.411) (0.410) (0.059) (0.081) 

Workplace type: Workplace type: 
Home-based care Not home-based care 

Signed up for the career lattice 0.195 0.169 —0.029 -0.078 0.030 0.081 na na 242 
Berveen 10/27/20 20a (0.398) (0.377) (0.118) (0.052) 
12/31/2019 
Workplace retention 0.783 0.814 —0.012 —0.032 —0.032 —0.082 na na 233 

(0.414) (0.391) (0.122) (0.058) 

Lives in an urban area Lives in a rural area 

Signed up for the career lattice 0.186 0.154 0.017 0.048 0.095 0.261 na na 289 
SS AOE Ie (0.390) (0.363) (0.046) (0.100) 
12/31/2019 
Workplace retention 0.803 0.811 -0.025 -0.063 0.074 0.188 na na 280 

(0.399) (0.393) (0.051) (0.102) 


na is not applicable. 

Note: None of the effects was statistically significant at p < .10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were estimated. 
c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants that who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table C3. The impact of the sign-up email on increase in identification as a professional during the study 
period, 2018 and 2019 


Treatment (eroyniage)| Estimated 
group mean group mean impact 
Gilater-lae! Gielatel-lae! (Gielater-lae} Effect 
Outcome deviation) deviation) error’) size‘ 
Increase in identification as a professional 0.184 0.280 —0.130 —0.287 99 
(0.391) (0.454) (0.084) 


Note: None of the effects was statistically significant at p < .10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were 
estimated. 

c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Incentive randomization monetary incentive emails, additional analyses 


The following tables show the treatment-on-the-treated estimates (table C4), the group analyses for the 
monetary incentive emails (table C5), and the secondary outcomes (table C6). 


Table C4. Treatment-on-the-treated estimates of the monetary incentive emails on career lattice movement, 
college credit hours earned, community-based training hours recorded, and workplace retention during the 
study period, 2018 and 2019 


Treatment Control group Estimated 
group mean mean impact? 
(Gielatel-lae} (Giclatel-lae} (Gimlatel-lae} 
Outcome deviation) deviation) error’) 
Increased at least one step in the career 0.195 0.179 0.088 0.228 155 
lattice since 9/1/2018 (0.399) (0.386) (0.166) 
College credit hours earned 15.143 18.968 1.196 0.016 155 
between 9/1/2018 and 12/31/2019 (46.806) (75.882) (28.983) 
Total community-based training hours 42.458 32.229 41.183* 1.382 155 
recorded between 9/1/2018 and 
Workplace retention 0.813 0.855 —0.176 —0.498 151 
(0.392) (0.354) (0.161) 


* significant at p< .05. 

Note: Unless otherwise indicated, none of the effects was statistically significant at p < .10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were 
estimated. 

b. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

c. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


The monetary incentive provided at earlier-than-usual steps in the career lattice had detectable differential 
impacts for some groups on training hours recorded and workplace retention but no impact on career lattice 
movement or college credit hours earned. The study team examined the impact of the monetary incentive 
provided at earlier-than-usual steps in the career lattice for groups of participants defined by age ranges, primary 
language, education levels, race/ethnicity, type of workplace, workplace roles, workplace type, urban or rural 
location, or distance to the closest in-state public college or university. There were statistically significant impacts 
on total community-based training hours recorded and workplace retention for some groups but no detectable 
impacts for any groups on increasing a career lattice step or college credit hours earned (table C5). Treated 
participants who were age 35-54, who had an education level below associate degree, or identified as White had 
higher average training hours recorded than the control group (by about 20, 16, and 16 hours, respectively). 
Treated participants who lived in a rural area or who were located more than 20 miles from the nearest public in- 
state college also had higher average training hours recorded than the control group (by about 22 hours and 34 
hours, respectively). For workplace retention, treated participants who were age 55 or older or lived less than five 
miles from the nearest public in-state college were less likely to be retained at the same workplace (by 18 and 16 
percentage points, respectively). While these differences were individually statistically significant, adjusting for 
the large number of comparisons eliminates significance. 
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Table C5. Group analyses of the monetary incentive emails on main outcomes during the study period, 2018 and 2019 


Treatment 


Control 


Ageleyo) group Estimated Estimated Estimated 
mean mean impact? impact? impact? 
(Gielareklac} (Giclateklae} (Gielarekyae} (Gieclareklae} (Gielateklac} 
Outcome deviation) deviation) error?) Effect size° error?) Effect size° error’) Effect size° sized 
Age group: younger than 35 Age group: 35-54 Age group: 55 plus and older 

Increased at least one step in the 0.195 0.179 0.020 0.051 0.066 0.171 —0.061 —0.158 155 
career lattice since 9/1/2018 (0.399) (0.386) (0.106) (0.076) (0.119) 
College credit hours earned 15.143 18.968 9.419 0.124 2.933 —0.039 -3.499 —-0.046 155 
between 9/1/2018 and 
12/31/2019 (46.806) (75.882) (16.463) (13.340) (15.076) 
Total community-based training 42.458 32.229 15.906 0.534 20.065t 0.673 —6.807 0.228 155 
hours recorded between 

48.034 29.797 13.200 10.263 7.905 
9/1/2018 and 12/31/2019 ( ) ( ) ( ) ( ) ( ) 
Workplace retention 0.813 0.855 —0.146 -0.413 0.026 0.072 -0.184t -0.519 151 

(0.392) (0.354) (0.117) (0.071) (0.099) 

Primary language: English Primary language: Non-English 

Increased at least one step in the 0.195 0.179 0.063 0.163 —0.123 —0.317 na na 155 
career lattice since 9/1/2018 (0.399) (0.386) (0.071) (0.139) 
College credit hours earned 15.143 18.968 7.671 0.101 —33.149 -0.437 na na 155 
between 9/1/2018 and 
12/31/2019 (46.806) (75.882) (9.999) (37.358) 
Total community-based training 42.458 32.229 12.048 0.404 23.128 0.776 na na 155 
hours recorded between 
9/1/2018 and 12/31/2019 (48.034) (29.797) (7.927) (16.814) 
Workplace retention 0.813 0.855 —0.067 -0.188 -0.028 —0.078 na na 151 

(0.392) (0.354) (0.068) (0.160) 


Outcome 


Treatment 
Ageleyo) 
mean 

(Gielaleklac} 
deviation) 


Control 
Agelele) 
mean 

(Gielaletlac} 
deviation) 


Estimated 


Tan} ex-\el ta 
(Gieclaleklae} 
error?) 


Effect size° 


Workplace role: 


Estimated 
Tan) ex-\el ta 
(Gielaleklae} 


error?) Effect size® 


Workplace role: 


Estimated 
Tan} ex-\el ta 
(Gielaleklae} 
error?) Effect size® 


Workplace role: 


Director/Owner Lead Teacher/Teacher Aide/Other 

Increased at least one step in the 0.195 0.179 0.054 0.139 0.113 0.294 —0.009 —0.023 155 
career lattice since 9/1/2018 (0.399) (0.386) (0.089) (0.096) (0.107) 
College credit hours earned 15.143 18.968 15.253 0.201 0.311 0.004 —16.541 -0.218 155 
between 9/1/2018 and 
12/31/2019 (46.806) (75.882) (11.211) (12.962) (25.571) 
Total community-based training 42.458 32.229 12.200 0.409 18.340 0.616 12.661 0.425 155 
hours recorded between 
Workplace retention 0.813 0.855 —0.019 —0.053 —0.103 —0.291 —0.138 —0.390 151 

(0.392) (0.354) (0.072) (0.110) (0.125) 

Education level: Education level: 
associate degree or higher below associate degree 

Increased at least one step in the 0.195 0.187 0.002 0.004 0.047 0.120 na na 152 
career lattice since 9/1/2018 (0.399) (0.392) (0.093) (0.074) 
College credit hours earned 15.143 19.727 10.911 0.141 —5.739 -0.074 na na 152 
between 9/1/2018 and (46.806) (77.307) (13.409) (14.199) 
12/31/2019 
Total community-based training 42.458 32.218 7.852 0.260 15.986t 0.528 na na 152 
hours recorded between 
9/1/2018 and 12/31/2019 (48.034) (30.254) (8.629) (8.414) 
Workplace retention 0.813 0.849 —0.053 —0.148 —0.057 —0.158 na na 148 

(0.392) (0.360) (0.097) (0.071) 


Treatment 
group 
mean 

(Gieclaleklac} 


Outcome deviation) 


Control 
Agelele) 
mean 

(Gielaletlac} 
deviation) 


Estimated 


Tan} ex-\el ta 
(Gieclaleklac} 
error?) Effect size® 


Race/ethnicity: White 


Estimated 
Tan} ex-\el ta 
(Gieclaleklae} 
error?) Effect size® 


Race/ethnicity: Other than White 


Estimated 
Tan} ex-\el ta 
(Gieclaleklae} 
error?) Effect size® 


size 


Increased at least one step in the 0.195 0.187 0.070 0.179 —0.069 —0.176 na na 152 
career lattice since 9/1/2018 (0.399) (0.392) (0.076) (0.092) 
College credit hours earned 15.143 19.727 10.981 0.142 —20.925 -0.271 na na 152 
between 9/1/2018 and 
12/31/2019 (46.806) (77.307) (10.755) (16.400) 
Total community-based training 42.458 31.328 16.166t 0.557 9.268 0.320 na na 152 
hours recorded between 
9/1/2018 and 12/31/2019 (48.034) (28.998) (9.142) (8.694) 
Workplace retention 0.813 0.851 —0.003 —0.008 —0.126 —0.352 na na 149 

(0.392) (0.358) (0.062) (0.110) 

Workplace type: Workplace type: 
Home-based care Not home-based care 

Increased at least one step in the 0.200 0.167 0.082 0.218 0.034 0.090 na na 142 
career lattice since 9/1/2018 (0.403) (0.375) (0.115) (0.071) 
College credit hours earned 16.657 19.160 13.812 0.176 —4,335 -0.055 na na 142 
between 9/1/2018 and 

48.861 78.274 12.4 15.014 
12/31/2019 (seer) UB ie) eed) ot) 
Total community-based training 43.564 32.998 16.273 0.529 13.572 0.441 na na 142 
hours recorded between 
Workplace retention 0.797 0.857 —0.001 —0.002 -0.107 —0.303 na na 139 

(0.405) (0.352) (0.080) (0.079) 


Treatment 
group 
mean 

(Gieclaleklac} 


Outcome deviation) 


Control 
group Estimated 
mean Tan} ex-\el ta 
(Gielaletlac} (Gielaleklac} 
deviation) error?) 


Effect size° 


Lives in an urban area 


Estimated 
Tan} ex-\el ta 
(Gielaleklae} 
error?) Effect size® 


Lives in a rural area 


Estimated 
Tan} ex-\el ta 
(Gielaleklac} 
error?) Effect size® 


Increased at least one step in the 0.195 0.179 0.029 0.075 0.033 0.085 na na 155 
career lattice since 9/1/2018 (0.399) (0.386) (0.073) (0.095) 
College credit hours earned 15.143 18.968 -5.361 -0.071 14.240 0.188 na na 155 
between 9/1/2018 and 

46. 75.882 11.54 15. 
12/31/2019 ACN) (WEY EZ.) (11.543) (15.359) 
Total community-based training 42.458 32.229 10.584 0.355 22.260* 0.747 na na 155 
hours recorded between 
9/1/2018 and 12/31/2019 (48.034) (29.797) (9.237) (11.115) 
Workplace retention 0.813 0.855 -0.107 -0.302 0.054 0.152 na na 151 

(0.392) (0.354) (0.073) (0.078) 

Distance to college: Distance to college: Distance to college: 
less than 5 miles 5-20 miles more than 20 miles 

Increased at least one step in the 0.189 0.189 0.036 0.091 —0.015 —0.037 0.125 0.316 148 
Rareen aioe since pt) 2008 (0.394) (0.394) (0.080) (0.094) (0.181) 
College credit hours earned 15.757 19.993 10.785 0.139 —2.690 -0.035 —29.202 -0.375 148 
between 9/1/2018 and 

47. 77. 12.184 13.771 ale 
Sa/EE Abie (47.655) (77.800) (12.184) (13.771) (31.805) 
Total community-based training 42.949 31.414 7.160 0.242 20.630 0.697 33.516T 1.133 148 
hours recorded between 
9/1/2018 and 12/31/2019 (48.591) (29.592) (7.654) (13.550) (17.949) 
Workplace retention 0.806 0.861 —0.157* —0.451 0.025 0.072 —0.098 —0.280 144 

(0.399) (0.348) (0.077) (0.091) (0.152) 


Tt Significant at p < .10; * significant at p < .05. 
na is not applicable. 


Note: None of the effects was statistically significant at p < .01. 

a. Estimated using a linear probability model with an indicator for treatment. 
b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were estimated. 
c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 
d. Refers to the sample size for the regression analysis and includes both treatment and control group participants that who had data for the outcome. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table C6. The impact of the monetary incentive emails on increase in identification as a professional and 
motivation to take a college course or earn a degree during the study period, 2018 and 2019 


Treatment erolniage)| 
group fAgeyeyo) Estimated 
mean mean impact® 
(Gielaler-lae} Giclalek-iae! (Gielater-lae} 
Outcome deviation) deviation) error’) 
Increase in identification as a professional 0.184 0.162 0.066 0.175 86 
(0.391) (0.374) (0.088) 
Motivation to take a college course in the next year 0.481 0.308 0.083 0.177 91 
(0.505) (0.468) (0.098) 
Motivation to take a college course in the next five years 0.442 0.410 0.050 0.101 91 
(0.502) (0.498) (0.121) 
Motivation to earn a degree in the next five years 0.404 0.436 —0.001 —0.002 91 
(0.495) (0.502) (0.109) 


Note: Motivation outcomes were measured as whether respondents reported high or very high motivation. None of the effects was statistically significant 
at p<.10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were 
estimated. 

c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Incentive randomization automatic scholarship enrollment emails, additional analyses 


The tables below show the treatment-on-the-treated estimates (table C7), the group analyses for the automatic 
scholarship enrollment emails (table C8), and the secondary outcomes (table C9). 


Table C7. Treatment-on-the-treated estimates of the automatic scholarship enrollment emails on scholarship 
use, career lattice movement, college credit hours earned, and workplace retention during the study period, 
2018 and 2019 


Treatment (eoyniune)| 


group group Estimated 
mean mean Tan) oy-\ei me 
(standard (standard (standard Effect Sample 

Outcome deviation) deviation) error’) size size’ 
Used a community college scholarship 0.063 0.026 0.119 0.748 158 

(0.244) (0.159) (0.125) 
Increased at least one step in the career lattice since 0.100 0.179 —0.325 0.842 158 
es (0.302) (0.386) ~— (0.225) 
College credit hours earned between 9/1/2018 and 16.038 18.968 —9.680 0.128 158 
12/31/2019 (41.755) (75.882) (30.757) 
Workplace retention 0.838 0.855 0.056 0.157 156 


(0.371) (0.354) (0.215) 


Note: None of the effects was statistically significant at p < .10. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Estimated impact were adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how 
standard errors were estimated. 

c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Scholarship emails had a differential impact on career lattice movement for some groups but not on scholarship 
use, college credit hours earned, or workplace retention. The scholarship emails did not have a detected 
differential impact by groups of participants defined by age ranges, primary language, education levels, 
race/ethnicity, type of workplace, workplace roles, workplace type, urban or rural location, or distance to the 
closest in-state public college or university on scholarship use, college credit hours, or workplace retention, but 
they did have an impact on career lattice movement (table C8). Treated participants who identified English as 
their primary language, had an education level below associate degree, or lived in an urban area were less likely 
to move up a step than the control group (by 12, 11, and 12 percentage points, respectively). Treated participants 
who were younger than age 35 or who were age 55 or older at baseline were less likely to increase a career lattice 
step than control participants in those groups (by 13 and 20 percentage points, respectively). For distance to 
college, treated participants who lived 5-20 miles from the nearest public in-state college were less likely to 
increase at least one step compared with the control group (by 13 percentage points). Again, adjusting for the 
large number of comparisons eliminated the statistical significance of these differences. 
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Outcome 


Treatment 
Ageleyo) 
mean 

(Gielarekyae} 
deviation) 


eroyahane)| 
group 
mean 
(Giclaleklae} 
deviation) 


Group 1 


Estimated 
impact? 
(Gielareklae} 


lanelan) Effect size° 


Age group: younger than 35 


Table C8. Group analyses of the impact of the automatic scholarship enrollment emails on main outcomes during the study period, 2018 and 2019 


Estimated 
impact? 
(Gieclareklae} 


error?) 


Age group: 35-54 


Effect size‘ 


Group 3 


Estimated 
impact? 

(Giclarerlae) Sample 
error’) Effect size° sized 


Age group: 55 and older 


Used a community college 0.063 0.026 0.033 0.207 0.002 0.013 0.143 0.902 158 
Snoltehlp (0.244) (0.159) (0.052) (0.045) (0.100) 
Increased at least one step in the 0.100 0.179 —0.132t -0.342 -0.026 —0.067 —0.198* -0.513 158 
Career lattice cince 9/1/2018 (0.302) (0.386) (0.071) (0.074) (0.085) 
College credit hours earned 16.038 18.968 8.746 0.115 -9.399 —0.124 —4.146 —-0.055 158 
between 9/1/2018 and 
12/31/2019 (41.755) (75.882) (11.357) (11.936) (12.057) 
Workplace retention 0.838 0.855 —0.019 —0.052 0.060 0.170 —0.088 —0.248 156 
(0.371) (0.354) (0.100) (0.076) (0.127) 
Primary language: English Primary language: Non-English 
Used a community college 0.063 0.026 0.027 0.170 0.048 0.301 na na 158 
scholarship (0.244) (0.159) (0.034) (0.107) 
Increased at least one step in the 0.100 0.179 —0.122* -0.315 0.087 0.226 na na 158 
career lattice since 9/1/2018 (0.302) (0.386) (0.060) (0.155) 
College credit hours earned 16.038 18.968 4.436 0.058 —34.253 -0.451 na na 158 
between 9/1/2018 and 
12/31/2019 (41.755) (75.882) (7.834) (27.600) 
Workplace retention 0.838 0.855 —0.008 —0.022 0.116 0.326 na na 156 
(0.371) (0.354) (0.066) (0.133) 


Outcome 


Treatment 
Ageleyo) 
mean 

(Gielaleklac} 
deviation) 


(rola age)| 
Agelele) 
mean 

(Gielaletlac} 
deviation) 


Group 1 


Estimated 


Tan} ex-\el ta 
(Gieclaleklae} 
error?) Effect size® 


Workplace role: 


Group 2 


Estimated 
Tan} ex-\el ta 
(Gieclaleklae} 


error?) Effect size® 


Workplace role: 


Group 3 


Estimated 
Tan} ex-\el ta 
(Gielaleklae} 


error?) Effect size® 


Workplace role: 


Director/Owner Lead Teacher/Teacher Aide/Other 

Used a community college 0.063 0.026 0.003 0.020 0.051 0.323 0.068 0.425 158 
senolarship (0.244) (0.159) (0.058) (0.054) (0.088) 
Increased at least one step in the 0.100 0.179 -0.123 —0.318 0.028 0.072 -0.113 -0.292 158 
Gareer lattice since 9/1/2018 (0.302) (0.386) (0.088) (0.083) (0.073) 
College credit hours earned 16.038 18.968 —4.106 —0.054 —9.838 —0.130 12.821 0.169 158 
between 9/1/2018 and 
12/31/2019 (41.755) (75.882) (14.613) (10.145) (14.728) 
Workplace retention 0.838 0.855 0.085 0.240 —0.044 —0.124 0.018 0.050 156 

(0.371) (0.354) (0.093) (0.089) (0.119) 

Education level: Education level: 
associate degree or higher below associate degree 

Used a community college 0.063 0.027 0.069 0.428 0.013 0.078 na na 155 
scholarship (0.244) (0.162) (0.062) (0.038) 
Increased at least one step in the 0.100 0.187 —0.029 -0.075 -0.111t -0.282 na na 155 
career lattice since 9/1/2018 (0.302) (0.392) (0.085) (0.065) 
College credit hours earned 16.038 19.727 5.462 0.071 -7.023 —0.091 na na 155 
between 9/1/2018 and 
12/31/2019 (41.755) (77.307) (10.669) (12.232) 
Workplace retention 0.838 0.849 —0.018 —0.050 0.040 0.112 na na 153 

(0.371) (0.360) (0.093) (0.071) 


Treatment 
group 
mean 

(Gielaleklac} 


Outcome deviation) 


(rola age)| 
Agelele) 
mean 

(Gielaletlac} 
deviation) 


Group 1 


Estimated 


Tan} ex-\el ta 
(Gielaleklae} 
error?) 


Effect size° 


Race/ethnicity: White 


Group 2 


Estimated 
Tan} ex-\el ta 
(Gielaleklac} 


error?) Effect size® 


Race/ethnicity: Other than White 


Group 3 


Estimated 
Tan} ex-\el ta 
(Gieclaleklac} 


error?) Effect size® 


Used a community college 0.064 0.027 0.025 0.156 0.030 0.185 na na 153 
scholarship (0.247) (0.162) (0.040) (0.062) 
Increased at least one step in the 0.103 0.187 -0.101 -0.257 —-0.056 -0.142 na na 153 
career lattice since 9/1/2018 (0.305) (0.392) (0.066) (0.090) 
College credit hours earned 16.449 19.727 7.085 0.092 —21.614 —0.280 na na 153 
between 9/1/2018 and 
12/31/2019 (42.212) (77.307) (9.416) (13.771) 
Workplace retention 0.833 0.851 0.016 0.045 0.024 0.067 na na 152 

(0.375) (0.358) (0.073) (0.098) 

Workplace type: Workplace type: 
Home-based care Not home-based care 
Used a community college 0.068 0.028 0.026 0.155 0.038 0.232 na na 146 
holarshi 

eee (0.253) (0.165) (0.056) (0.047) 
Increased at least one step in the 0.108 0.167 -0.116 —0.310 -0.018 -0.049 na na 146 
career lattice since 9/1/2018 (0.313) (0.375) (0.087) (0.066) 
College credit hours earned 13.446 19.16 —2.731 —0.035 —4.490 —0.057 na na 146 
between 9/1/2018 and 
12/31/2019 (34.985) (78.274) (15.007) (8.888) 
Workplace retention 0.851 0.857 0.066 0.188 -0.017 -0.049 na na 144 

(0.358) (0.352) (0.087) (0.077) 


Outcome 


Treatment 
Ageleyo) 
mean 

(Gieclaleklac} 
deviation) 


Group 1 


(roy ah aae)| 
group Estimated 


mean Tan} ex-\el ta 


(Gielalet-lac} (Gieclaleklae} 


deviation) error?) Effect size° 


Lives in an urban area 


Group 2 


Estimated 
Tan} ex-\el ta 
(Gielaleklac} 
error?) Effect size® 


Lives in a rural area 


Group 3 


Estimated 
Tan} ex-\el ta 
(Gieclaleklac} 
error?) Effect size® 


Used a community college 0.063 0.026 0.003 0.016 0.103 0.650 na na 158 
scholarship (0.244) (0.159) (0.035) (0.068) 
Increased at least one step in the 0.100 0.179 —0.117t -0.304 0.001 0.001 na na 158 
career lattice since 9/1/2018 (0.302) (0.386) (0.063) (0.100) 
College credit hours between 16.038 18.968 —1.063 —0.014 —6.226 —0.082 na na 158 
9/1/2018 and 12/31/2015 (41.755) (75.882) (9.217) (14.642) 
Retention 0.838 0.855 0.023 0.065 —-0.007 —-0.020 na na 156 
(0.371) (0.354) (0.070) (0.094) 
Distance to college: Distance to college: Distance to college: 
less than 5 miles 5-20 miles more than 20 miles 
Used a community college 0.063 0.027 0.041 0.251 0.029 0.179 —0.018 —0.110 153 
scholarship (0.245) (0.163) (0.051) (0.046) (0.060) 
Increased at least one step in the 0.089 0.189 -0.053 -0.133 —0.132* -0.335 -0.139 —0.352 153 
Career latte eine) 3/1/2018 (0.286) (0.394) (0.081) (0.063) (0.150) 
College credit hours between 16.241 19.993 2.152 0.028 -8.461 —0.109 7.685 0.099 153 
9/1/2018 and 12/31/2013 (41.982) (77.800) (10.048) (8.664) (42.459) 
Retention 0.835 0.861 0.001 0.004 —0.045 -0.129 0.216 0.619 151 
(0.373) (0.348) (0.079) (0.080) (0.130) 


T Significant at p < .10; * significant at p < .05. 


na is not applicable. 


Note: None of the effects was statistically significant at p < .01. 

a. Estimated using a linear probability model with an indicator for treatment. 
b. Adjusted using randomization inference tests to account for the use of rerandomization. See appendix B for a description of how standard errors were estimated. 
c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 
d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Table C9. The impact of automatic scholarship enrollment emails on increase in identification as a 
professional, motivation to take a college course in the next year and the next five years, and motivation to 
earn a degree in the next five years during the study period, 2018 and 2019 


Treatment (ol aliae)| 


fagey eyo) Age} eyo) Estimated 
mean mean impact? 
(Gielalerlae} (standard (standard Effect 

Outcome deviation) deviation) error’) size‘ 

Increase in identification as a professional 0.018 0.162 —0.105 —0.281 81 
(0.390) (0.374) (0.103) 

Motivation to take a college course in the next year 0.542 0.308 0.319** 0.683 87 
(0.504) (0.468) (0.119) 

Motivation to take a college course in the next five years 0.542 0.410 0.201t 0.403 87 
(0.504) (0.498) (0.109) 

Motivation to earn a degree in the next five years 0.563 0.436 0.191t 0.380 87 


(0.501) (0.502) (0.105) 


T Significant at p < .10; ** significant at p< .01. 

a. Estimated using a linear probability model with an indicator for treatment. 

b. Standard errors for the estimated impact were adjusted using randomization inference tests to account for the use of rerandomization. See appendix B 
for a description of how standard errors were estimated. 

c. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

d. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 

Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Survey results: Additional analyses and findings 

This section reports descriptive findings from the participant surveys, which provide additional context about 
treatment group participants’ perceived receipt of and responses to the intervention emails. This is followed by 
general survey questions about factors motivating professional development among early childhood education 
workforce members. Findings from responses to the follow-up surveys for each intervention should be viewed 
with caution because of small sample sizes among the treatment group respondents and uncertainty whether 
respondents accurately represent the original study sample. The study team was unable to conduct a nonresponse 
bias analysis because few background characteristics were correlated with outcomes; thus, the study team could 
not diagnose the extent of nonresponse bias. 


Small numbers of respondents indicated that the email information motivated behavior change. Among a very 
small number of treatment group participants responding to follow-up survey questions (about 20-50 percent of 
respondents depending on the treatment group—only a small number of respondents were eligible to answer 
this question on the survey), 50-80 percent reported that the email they received motivated them to sign up for 
the career lattice, complete training hours or college coursework, or access the scholarship they had been enrolled 
in. This tentatively suggests that that the emails motivated some recipients in positive ways, although those 
impacts were not detected statistically. 


Small numbers of survey respondents indicated that a lack of information, qualifying educational activities, and 
time were reasons for not signing up for the career lattice or applying for an education award or accessing the 
scholarship. Among the small number of participants in the sign-up randomization treatment group who 
responded to a follow-up survey question on why they did not sign up for the career lattice (approximately 40 
percent of the treatment group respondents), the most common reason for not signing up for the career lattice 
was not being sure how to sign up (less than 50 percent of respondents mentioned this), followed by not knowing 
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about the steps program. This provides suggestive evidence that some participants in the sign-up treatment group 
might not have read the emails. 


Some survey respondents from the incentive randomization reported that they did not apply for an education 
award or access the scholarship because of a lack of qualifying professional development activities, plans to pursue 
college courses, or a lack of time. Among the few treatment group participants in the incentive randomization 
who responded to follow-up survey questions (30—60 percent of respondents depending on the treatment group), 
the most common reason given for not applying for an education award or accessing a scholarship was not 
qualifying for an education award during the study period (less than 50 percent of respondents) or not having 
time to take college courses (about 60 percent). 


Among respondents in the sign-up randomization treatment group who said they signed up for the career lattice, 
a majority indicated that they did so to be recognized for their education achievements, to plan for professional 
development, and because their employer required it. Among a small number of respondents in the incentive 
randomization treatment groups (50-80 percent of participants depending on the group), the majority indicated 
that education awards and scholarships offered through Oregon Center for Career Development in Childhood Care 
and Education (OCCD) motivated them to a moderate or large extent to continue education or training. 


Follow-up survey respondents reported mixed recollections of receiving the emails. Many treatment group 
participants across interventions reported that they either did not receive or did not know if they received an 
email about signing up for the career lattice, monetary education award information, or automatic scholarship 
enrollment (depending on intervention group). However, these findings should be interpreted with caution due 
to the small sample sizes in each group. Less than half the sign-up treatment group (44 percent) and the automatic 
scholarship enrollment treatment group (35 percent) reported in the follow-up survey that they had received an 
email about their respective interventions from OCCD (table C10). About half (53 percent) of the monetary 
incentive treatment group who responded to the follow-up survey reported receiving the email. This implies that 
although administrative records indicate participants opened the email, many recipients did not recall the email 
or the content that was specific to this study. Participants might not have read or fully engaged with email content 
as intended through the intervention, which could in turn help explain the null findings of the study. 


Table C10. Follow-up survey responses by randomized treatment group regarding receipt of an intervention 
email, 2019 (percent) 


Incentive 


Incentive randomization 
randomization automatic 
Sign-up monetary scholarship 
randomization incentive lala) ifaatslale 
treatment treatment treatment 
group group group 
Survey question response (n=55) (n=51) (n = 48) 
Yes, received an email about the intervention from OCCD since August 2018 44 53 35 
No, did not receive an email about the intervention from OCCD 29 25 
soa PEST SSSRSRSTSPRT TSS 47? 
Don’t know if received an email about the intervention from OCCD 27 40 


OCCD is Oregon Center for Career Development in Childhood Care and Education. 
a. Results were combined to protect respondent privacy due to small respondent numbers. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2019. 
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A majority of survey respondents at baseline reported that they planned to continue their formal education and 
planned to participate in non-credit-bearing professional development (table C11). Additionally, most 
respondents in the incentive randomization reported at least a moderate level of motivation to take a college 
course or earn a degree in the near future. More respondents reported a high or very high level of motivation to 
take a college course in the next five years compared with the next year, indicating that the timeline for planning 
to return to college is likely closer to five years than one year for many respondents. 


Table C11. Baseline survey responses by randomized trial regarding continuing education or professional 
development, 2018 (percent) 


Sign-up Incentive 
randomization randomization 
Survey question and response a s)) (n = 207) 
Plan to continue formal education (n = 244) (n = 207) 
Yes 59 62 
No 14 13 
Don’t know 27 26 
Plan to participate in professional development for which will not (n = 245) (n = 207) 
receive college course credit 
Yes 59 76 
No 14 8 
Don’t know 27 16 
Level of motivation to take a college course in the next year na (n = 193) 
Very low 15 
Low 10 
Moderate 33 
High 19 
Very high 23 
Level of motivation to take a college course in the next five years na (n = 192) 
Very low 13 
Low 8 
Moderate 29 
High 24 
Very high 26 
Level of motivation to earn a higher education degree or early na (n = 193) 
childhood education-related certification in the next five years 
Very low 13 
Low 9 
Moderate 24 
High 23 
Very high 31 


na is not applicable because the question was not asked of the sign-up randomization participants. 
Note: Percentages might not sum to 100 because of rounding. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2019. 
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Most participants (more than 80 percent in both trials) reported at baseline that personal growth was a motivation 
to continue their education and training (table C12). Getting a job was a motivator for more participants in the 
sign-up randomization (27 percent) than in the incentive randomization (12 percent), whereas licensing or 
workplace requirements were noted as motivators for more participants in the incentive randomization (more 
than half) than in the sign-up randomization (less than half). Fewer respondents (11 percent for sign-up and 23 
percent for incentive) reported that scholarship money was a motivator. 


Table C12. Baseline survey responses by randomization trial regarding motivating factors to continue 
education and training, 2018 (percent) 


Sign-up Incentive 
randomization randomization 


Survey question and response (n = 198) (ae kt) 


What motivates you to continue your education and training? 


Personal growth 88 84 
Meet Office of Child Care licensing requirements 46 66 
My workplace requires me to continue education and training 46 52 
Receive a promotion/pay raise 35 42 
Get a job 27 12 
Scholarship money 11 23 
Spark requirements? 6 22 
Other 9 10 


Note: Respondents could select multiple responses. 
a. Spark refers to Oregon’s quality rating and improvement system. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2019. 


More than half (56 percent) of incentive randomization participants responding to the baseline survey question 
about factors affecting their decision to continue their education ranked cost as the most important factor and 82 
percent ranked it as one of their top three factors (table C13). Time was the second most highly ranked factor, 
followed by family commitments. Distance was ranked in the top three factors for about a quarter (27 percent) of 
participants. 


Table C13. Baseline survey responses for incentive randomization participants regarding the most important 
factors affecting the decision to continue education, 2018 (percent) 
Ranked 1 Ranked 1 or 2 Ranked 1, 2, or 3 


Survey question and response (n = 188) (n = 188) (n = 188) 


What factors affect your decision to continue your education? 
(Responses ranked with the most important factor ranked first; 
not all options had to be ranked) 


Cost 56 71 82 
Time 27 61 81 
Family commitments 8 29 51 
Distance 1 15 27 
Language barrier 4 9 12 
Difficulty of subject matter 1 3 8 
Other 4 5 7 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2019. 
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Among the participants responding to the follow-up survey questions, about half across both the sign-up and 
incentive randomizations believed that a step in the career lattice helped them a lot to feel more professional or 
to plan for their professional development, and 9-15 percent felt it did not help at all (table C14). About 34-38 
percent indicated that a step helped them connect with others a lot, and 29-41 percent felt it did not help them 
connect with others at all. 


Table C14. Follow-up survey responses by randomization trial regarding benefits of having a step, 2019 
(percent) 


Sign-up Incentive 
re-dateeanly4-li(elammmme-lalele)galrZ-haleya) 
Survey question and response (asks) (n = 137) 
How much has a step on the Oregon Registry helped you feel more professional? n=32 n=137 
Helped a lot 44 53 
Helped a little 47 32 
Did not help at all 9 15 
How much has a step on the Oregon Registry helped you plan your professional development? n=33 n=135 
Helped a lot 45 49 
Helped a little 40 36 
Did not help at all 15 15 
How much has a step on the Oregon Registry helped you connect to others who care about n=32 n=136 
children and families? 
Helped a lot 38 34 
Helped a little 22 38 
Did not help at all 41 29 


Note: The sign-up randomization sample is small because only those who said they had registered for the career lattice were asked these questions. 
Percentages might not sum to 100 because of rounding. 
Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2019. 
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